diff --git a/.gitignore b/.gitignore index 542bc5035..7bab478ff 100644 --- a/.gitignore +++ b/.gitignore @@ -281,3 +281,17 @@ lance_compare/ # Linting binaries downloaded by local pre-commit hooks shellcheck*/ + +# --- vllm-embedding --------------------------------------------------------- +# Measurement artifacts are the substance of this fork, not build output: every +# published figure is traceable to a file below. They are tracked despite the +# blanket `*.json` and `results/` rules above. Binary captures (.npz, .onnx, +# .wav, .npy) stay ignored. +!benchmarks/edge_harness/results/ +!benchmarks/edge_harness/results/** +!baselines/ +!baselines/** +benchmarks/edge_harness/results/**/*.npz +benchmarks/edge_harness/results/**/*.onnx +benchmarks/edge_harness/results/**/*.npy +benchmarks/edge_harness/results/**/*.wav diff --git a/NOTICE b/NOTICE index ae1967a26..97661ad79 100644 --- a/NOTICE +++ b/NOTICE @@ -18,6 +18,17 @@ README.md; the substantive ones are: vllm_omni/platforms/cpu/, vllm_omni/patch.py CPU platform and CPU-backend patches benchmarks/edge_harness/ - measurement tooling (original to this fork) + measurement tooling and results (original to this fork) + baselines/, docs/engines/, docs/analysis/ + benchmark results and written analyses (original to this fork) + +vLLM itself is not modified and no vLLM source is redistributed here. This fork +changes vLLM's CPU behaviour through its own extension points at runtime +(vllm_omni/patch.py); the pinned upstream versions are recorded in +docs/edge/environment.md. + +Baseline measurements reference llama.cpp (https://github.com/ggml-org/llama.cpp, +MIT). No llama.cpp source is redistributed; only benchmark output produced with +it, under baselines/llama_cpp/. Upstream's README is preserved at docs/README.vllm-omni.md. diff --git a/README.md b/README.md index a52c985e8..c7ffd599b 100644 --- a/README.md +++ b/README.md @@ -12,14 +12,19 @@ measured came from the code path you thought it did. Upstream's own README is preserved at [docs/README.vllm-omni.md](docs/README.vllm-omni.md). -## What this fork adds +## What's here -| area | change | +| area | where | |---|---| -| 4-bit CPU quantization | `cpu_int4` — a W4A16 path built on PyTorch's tinygemm kernel, with the prefill path reading the kernel's own weight layout instead of a duplicate | -| Memory | RoPE tables sized to the servable context; weights repacked in blocks; KV budget that is actually honoured on CPU | +| 4-bit CPU quantization | `vllm_omni/model_executor/layers/quantization/cpu_int4.py` — W4A16 on PyTorch's tinygemm kernel, prefill reading the kernel's own weight layout instead of a duplicate | +| Memory | RoPE tables sized to the servable context; weights repacked in blocks; a KV budget that is actually honoured on CPU | +| CPU platform + vLLM patches | `vllm_omni/platforms/cpu/`, `vllm_omni/patch.py` — vLLM itself is **unmodified**; see [docs/edge/environment.md](docs/edge/environment.md) | | Device routing | `vllm_omni/edge/weight_path.py` picks the weight format per device class, from measurements rather than assumption | -| Measurement | `benchmarks/edge_harness/` — the benchmark, profiling, memory-attribution and fidelity tooling every number here was produced with | +| Measurement harness | `benchmarks/edge_harness/` — the grid runner, benchmarks, memory ledger, fidelity metric and agent suite every number here came from | +| Raw results | `benchmarks/edge_harness/results/` — 100+ artifacts backing the published figures, each with provenance | +| Baselines | `baselines/` — llama.cpp runs and the comparison scripts | +| Engine analyses | `docs/engines/` — llama.cpp, ExecuTorch, MNN, ncnn, vLLM, vLLM-Omni | +| Study documents | `docs/analysis/` — the full reports these summaries distil | ## Headline results @@ -47,6 +52,11 @@ These overlap and **must not be added** — see [docs/edge/results.md](docs/edge | `cpu_gemm_wna16` fallback | 32.3 tok/s | — | | llama.cpp Q4_K_M (reference) | 90.0 tok/s | 1121 tok/s | +Decode is llama.cpp's; prefill is ours by ~2×. Neither engine is +bandwidth-bound — both sit at 33–39% of the measured 222 GB/s ceiling — so the +decode gap is the execution model, not the kernels. See +[docs/edge/comparisons.md](docs/edge/comparisons.md). + ## Why the numbers here are worth reading This project has retracted five conclusions, every one of them a measurement @@ -78,9 +88,17 @@ print(llm.generate(['Hello'], SamplingParams(max_tokens=32))[0].outputs[0].text) " ``` -See [docs/edge/cpu-4bit.md](docs/edge/cpu-4bit.md) for building a checkpoint and -choosing between the 4-bit paths, and [ROADMAP.md](ROADMAP.md) for what is -measured, what is next, and what was tried and abandoned. +## Documentation + +| | | +|---|---| +| [docs/edge/results.md](docs/edge/results.md) | every measured figure, with spread and baseline | +| [docs/edge/methodology.md](docs/edge/methodology.md) | how to measure on a shared machine without fooling yourself — the five retractions and the rules they left | +| [docs/edge/cpu-4bit.md](docs/edge/cpu-4bit.md) | the two 4-bit paths, the packed layout as measured, building a checkpoint | +| [docs/edge/comparisons.md](docs/edge/comparisons.md) | against llama.cpp and the other engines considered | +| [docs/edge/environment.md](docs/edge/environment.md) | pins, machine, and why there is no vLLM patch series | +| [docs/edge/harness.md](docs/edge/harness.md) | running a comparison | +| [ROADMAP.md](ROADMAP.md) | what is next, with expected gains and how each would be verified — and what was rejected | ## Licence diff --git a/baselines/README.md b/baselines/README.md new file mode 100644 index 000000000..84354ec48 --- /dev/null +++ b/baselines/README.md @@ -0,0 +1,32 @@ +# Baselines + +Measurements of other engines and of unquantized references, kept so the +comparisons in [docs/edge/comparisons.md](../docs/edge/comparisons.md) can be +checked rather than taken on trust. + +## `llama_cpp/` + +llama.cpp is the reference for CPU inference of a quantized model. Pinned +commit and build flags are in +[docs/edge/environment.md](../docs/edge/environment.md); no llama.cpp source is +redistributed here, only output produced with it. + +| file | what | +|---|---| +| `llamacpp_q4km_t{4,8,16,32}.json` | Q4_K_M across thread counts | +| `llamacpp_grid_t*.json`, `llamacpp_node1_t*.json` | thread and NUMA-node sweeps | +| `ab_llamacpp_*.json` | interleaved A/B against vLLM on the same cores | +| `il_llamacpp_p{1,2,3}.json` | the interleaved passes behind the headline decode figure | +| `cmp_llamacpp.py` | the comparison driver | + +Both engines are pinned with `taskset` to the same physical cores and run +strictly one at a time; vLLM's prefix caching is disabled, because `llama-bench` +has no such cache and leaving it on would not be a comparison. + +## Quality references + +`quality_*.json` and `quality_cmp_*.json` hold the greedy-divergence comparison: +each build against its **own** bf16 output over 12 prompts, plus the +cross-engine roll-up. Read the caveats in +[docs/edge/comparisons.md](../docs/edge/comparisons.md) before quoting these — +the bit budgets are not matched, and the metric saturates. diff --git a/baselines/llama_cpp/ab_llamacpp_after.json b/baselines/llama_cpp/ab_llamacpp_after.json new file mode 100644 index 000000000..29c9eecb5 --- /dev/null +++ b/baselines/llama_cpp/ab_llamacpp_after.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T18:59:20Z", + "avg_ns": 580079289, + "stddev_ns": 22445145, + "avg_ts": 883.695051, + "stddev_ts": 34.173372, + "samples_ns": [ 550140411, 579685176, 611186497, 588372185, 571012179 ], + "samples_ts": [ 930.671, 883.238, 837.715, 870.197, 896.653 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T18:59:24Z", + "avg_ns": 2118395683, + "stddev_ns": 31881177, + "avg_ts": 60.434035, + "stddev_ts": 0.909662, + "samples_ns": [ 2110731151, 2152277901, 2078510078, 2149464130, 2100995155 ], + "samples_ts": [ 60.6425, 59.4719, 61.5826, 59.5497, 60.9235 ] + } +] diff --git a/baselines/llama_cpp/ab_llamacpp_before.json b/baselines/llama_cpp/ab_llamacpp_before.json new file mode 100644 index 000000000..c87a6990b --- /dev/null +++ b/baselines/llama_cpp/ab_llamacpp_before.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T18:50:11Z", + "avg_ns": 453009837, + "stddev_ns": 4617262, + "avg_ts": 1130.311918, + "stddev_ts": 11.483962, + "samples_ns": [ 459248279, 451575469, 456242077, 449875010, 448108352 ], + "samples_ts": [ 1114.87, 1133.81, 1122.21, 1138.09, 1142.58 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T18:50:14Z", + "avg_ns": 1475847924, + "stddev_ns": 9833345, + "avg_ts": 86.732884, + "stddev_ts": 0.578067, + "samples_ns": [ 1489621433, 1475111972, 1476264136, 1476396077, 1461846002 ], + "samples_ts": [ 85.9279, 86.7731, 86.7054, 86.6976, 87.5605 ] + } +] diff --git a/baselines/llama_cpp/ab_llamacpp_t24.json b/baselines/llama_cpp/ab_llamacpp_t24.json new file mode 100644 index 000000000..17fd842bf --- /dev/null +++ b/baselines/llama_cpp/ab_llamacpp_t24.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T18:44:52Z", + "avg_ns": 487743656, + "stddev_ns": 25108865, + "avg_ts": 1051.969118, + "stddev_ts": 54.409083, + "samples_ns": [ 484545979, 492513415, 522973237, 486232711, 452452942 ], + "samples_ts": [ 1056.66, 1039.57, 979.018, 1052.99, 1131.61 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T18:44:55Z", + "avg_ns": 1589326576, + "stddev_ns": 47473222, + "avg_ts": 80.594928, + "stddev_ts": 2.413871, + "samples_ns": [ 1529730792, 1557930160, 1636596682, 1585818829, 1636556420 ], + "samples_ts": [ 83.6749, 82.1603, 78.2111, 80.7154, 78.213 ] + } +] diff --git a/baselines/llama_cpp/cmp_llamacpp.py b/baselines/llama_cpp/cmp_llamacpp.py new file mode 100644 index 000000000..2a09bc3aa --- /dev/null +++ b/baselines/llama_cpp/cmp_llamacpp.py @@ -0,0 +1,106 @@ +"""Greedy-decode parity for llama.cpp against the same fp32 reference vLLM is +scored on. + +The point of the comparison is that "4-bit" is not one thing: llama.cpp's +Q4_K_M and our `cpu_int4` spend a similar number of bits per weight but pick +different grids, so the speed comparison is only meaningful next to a quality +comparison run on identical prompts. llama-server takes a prompt as raw token +ids, so both engines see exactly the token sequence the reference did. +""" +import argparse, json, subprocess, time, urllib.request + +import numpy as np + + +def post(url, payload, timeout=600): + req = urllib.request.Request( + url, data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=timeout) as r: + return json.loads(r.read()) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--gguf", required=True) + ap.add_argument("--ref", default="ref.json") + ap.add_argument("--ref-logits", default="ref_logits.npy") + ap.add_argument("--bin", default="/data/zhoutaichang/embedding_infer/llama.cpp/build/bin/llama-server") + ap.add_argument("--cores", default="56-87") + ap.add_argument("--threads", type=int, default=32) + ap.add_argument("--port", type=int, default=18099) + ap.add_argument("--ctx", type=int, default=4096) + ap.add_argument("--out", default="parity_llamacpp.json") + a = ap.parse_args() + + ref = json.load(open(a.ref)) + ref_logits = np.load(a.ref_logits) + n_new = len(ref["new_ids"]) + + proc = subprocess.Popen( + ["taskset", "-c", a.cores, a.bin, "-m", a.gguf, "--port", str(a.port), + "-t", str(a.threads), "-c", str(a.ctx), "--no-warmup", "--host", "127.0.0.1"], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + url = f"http://127.0.0.1:{a.port}" + try: + for _ in range(180): + try: + urllib.request.urlopen(url + "/health", timeout=2) + break + except Exception: + time.sleep(1) + else: + raise RuntimeError("llama-server did not become healthy") + + res = post(url + "/completion", { + "prompt": ref["prompt_ids"], "n_predict": n_new, + "temperature": 0.0, "top_k": 1, "n_probs": 20, + "cache_prompt": False, "post_sampling_probs": False, + }) + finally: + proc.terminate() + try: + proc.wait(timeout=30) + except subprocess.TimeoutExpired: + proc.kill() + + probs = res.get("completion_probabilities") or [] + ids = [p["id"] for p in probs] if probs and "id" in probs[0] else [] + n = min(n_new, len(ids)) if ids else 0 + match = sum(int(x == y) for x, y in zip(ref["new_ids"][:n], ids[:n])) + + rep = { + "engine": "llama.cpp", "gguf": a.gguf.split("/")[-1], + "ref_text": ref["text"], "llamacpp_text": res.get("content", ""), + "greedy_token_match": f"{match}/{n}" if n else "n/a", + "exact_match": bool(n) and ref["new_ids"][:n] == ids[:n], + "ref_top1": ref["top1"], + "llamacpp_first_token": ids[0] if ids else None, + "first_token_agree": bool(ids) and ids[0] == ref["top1"], + } + if probs: + import math + top = (probs[0].get("top_logprobs") or probs[0].get("top_probs") + or probs[0].get("probs") or []) + first = {} + for e in top: + if "id" not in e: + continue + first[int(e["id"])] = (float(e["prob"]) if "prob" in e + else math.exp(float(e["logprob"]))) + ref_top = np.argsort(-ref_logits)[:20] + ref_p = np.exp(ref_logits - ref_logits.max()) + ref_p /= ref_p.sum() + pairs = [(int(t), float(ref_p[t]), first[int(t)]) for t in ref_top if int(t) in first] + rep["top20_prob_overlap"] = f"{len(pairs)}/20" + if pairs: + d = [abs(r - v) for _, r, v in pairs] + rep["top20_prob_max_abs_diff"] = max(d) + rep["top20_prob_mean_abs_diff"] = sum(d) / len(d) + print(json.dumps(rep, indent=2, ensure_ascii=False)) + with open(a.out, "w") as f: + json.dump(rep, f, indent=2, ensure_ascii=False) + + +if __name__ == "__main__": + main() diff --git a/baselines/llama_cpp/il_llamacpp_p1.json b/baselines/llama_cpp/il_llamacpp_p1.json new file mode 100644 index 000000000..99cfb5d1d --- /dev/null +++ b/baselines/llama_cpp/il_llamacpp_p1.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T19:04:49Z", + "avg_ns": 489021600, + "stddev_ns": 23921490, + "avg_ts": 1048.939735, + "stddev_ts": 49.910229, + "samples_ns": [ 527122937, 485801934, 462083348, 490951261, 479148522 ], + "samples_ts": [ 971.31, 1053.93, 1108.03, 1042.87, 1068.56 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T19:04:52Z", + "avg_ns": 1570942064, + "stddev_ns": 20220055, + "avg_ts": 81.490699, + "stddev_ts": 1.061319, + "samples_ns": [ 1586700558, 1582528379, 1537750061, 1565679809, 1582051516 ], + "samples_ts": [ 80.6705, 80.8832, 83.2385, 81.7536, 80.9076 ] + } +] diff --git a/baselines/llama_cpp/il_llamacpp_p2.json b/baselines/llama_cpp/il_llamacpp_p2.json new file mode 100644 index 000000000..1c7fba04d --- /dev/null +++ b/baselines/llama_cpp/il_llamacpp_p2.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T19:14:20Z", + "avg_ns": 464927159, + "stddev_ns": 22146638, + "avg_ts": 1103.200971, + "stddev_ts": 51.267874, + "samples_ns": [ 453191976, 479254490, 448568607, 496875740, 446744984 ], + "samples_ts": [ 1129.76, 1068.33, 1141.41, 1030.44, 1146.07 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T19:14:23Z", + "avg_ns": 1422234679, + "stddev_ns": 12240631, + "avg_ts": 90.004494, + "stddev_ts": 0.767183, + "samples_ns": [ 1422285965, 1416723437, 1414194914, 1414611159, 1443357923 ], + "samples_ts": [ 89.996, 90.3493, 90.5109, 90.4842, 88.6821 ] + } +] diff --git a/baselines/llama_cpp/il_llamacpp_p3.json b/baselines/llama_cpp/il_llamacpp_p3.json new file mode 100644 index 000000000..42a8aec09 --- /dev/null +++ b/baselines/llama_cpp/il_llamacpp_p3.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T19:20:46Z", + "avg_ns": 456731353, + "stddev_ns": 6199988, + "avg_ts": 1121.174997, + "stddev_ts": 15.277320, + "samples_ns": [ 459406023, 462620142, 461507407, 450329050, 449794146 ], + "samples_ts": [ 1114.48, 1106.74, 1109.41, 1136.95, 1138.3 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 24, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T19:20:49Z", + "avg_ns": 1506056809, + "stddev_ns": 24193761, + "avg_ts": 85.007660, + "stddev_ts": 1.362252, + "samples_ns": [ 1478402259, 1532453276, 1530685126, 1494657866, 1494085519 ], + "samples_ts": [ 86.58, 83.5262, 83.6227, 85.6383, 85.6711 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_bench.json b/baselines/llama_cpp/llamacpp_bench.json new file mode 100644 index 000000000..c092a14d6 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_bench.json @@ -0,0 +1,1082 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:48:28Z", + "avg_ns": 5826194444, + "stddev_ns": 18499227, + "avg_ts": 87.879684, + "stddev_ts": 0.278992, + "samples_ns": [ 5805110833, 5811473500, 5847614162, 5841912112, 5824861616 ], + "samples_ts": [ 88.1981, 88.1016, 87.5571, 87.6425, 87.8991 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:49:03Z", + "avg_ns": 14610334471, + "stddev_ns": 123024936, + "avg_ts": 8.761414, + "stddev_ts": 0.072994, + "samples_ns": [ 14565270602, 14580640193, 14828037521, 14541752491, 14535971551 ], + "samples_ts": [ 8.78803, 8.77876, 8.6323, 8.80224, 8.80574 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:50:16Z", + "avg_ns": 2918362099, + "stddev_ns": 12286131, + "avg_ts": 175.443371, + "stddev_ts": 0.741092, + "samples_ns": [ 2925259382, 2929673132, 2898545311, 2923305305, 2915027369 ], + "samples_ts": [ 175.027, 174.764, 176.64, 175.144, 175.642 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:50:34Z", + "avg_ns": 7865646966, + "stddev_ns": 53771152, + "avg_ts": 16.273905, + "stddev_ts": 0.111300, + "samples_ns": [ 7791699373, 7865327825, 7937201692, 7889061755, 7844944189 ], + "samples_ts": [ 16.4277, 16.274, 16.1266, 16.225, 16.3162 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:51:13Z", + "avg_ns": 1491348299, + "stddev_ns": 5631768, + "avg_ts": 343.317407, + "stddev_ts": 1.294111, + "samples_ns": [ 1489819485, 1499678994, 1485448813, 1494052451, 1487741754 ], + "samples_ts": [ 343.666, 341.406, 344.677, 342.692, 344.146 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:51:22Z", + "avg_ns": 5197057974, + "stddev_ns": 84695512, + "avg_ts": 24.634512, + "stddev_ts": 0.398355, + "samples_ns": [ 5165558444, 5161453873, 5102450401, 5324858098, 5230969056 ], + "samples_ts": [ 24.7795, 24.7992, 25.086, 24.0382, 24.4697 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:51:48Z", + "avg_ns": 825028648, + "stddev_ns": 6429234, + "avg_ts": 620.614565, + "stddev_ts": 4.820950, + "samples_ns": [ 819600650, 819801745, 830087549, 822025580, 833627717 ], + "samples_ts": [ 624.694, 624.541, 616.802, 622.852, 614.183 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:51:53Z", + "avg_ns": 4431906380, + "stddev_ns": 123688081, + "avg_ts": 28.899578, + "stddev_ts": 0.810937, + "samples_ns": [ 4392305744, 4530404684, 4573176188, 4261345178, 4402300109 ], + "samples_ts": [ 29.1419, 28.2535, 27.9893, 30.0375, 29.0757 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:52:20Z", + "avg_ns": 2086923625, + "stddev_ns": 25271587, + "avg_ts": 245.366053, + "stddev_ts": 2.978325, + "samples_ns": [ 2055651221, 2066285152, 2110647408, 2110779760, 2091254587 ], + "samples_ts": [ 249.069, 247.788, 242.58, 242.564, 244.829 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:52:33Z", + "avg_ns": 5013971032, + "stddev_ns": 41743502, + "avg_ts": 25.530080, + "stddev_ts": 0.212086, + "samples_ns": [ 5039794469, 5070632551, 5009766482, 4977188073, 4972473585 ], + "samples_ts": [ 25.3979, 25.2434, 25.5501, 25.7173, 25.7417 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:52:58Z", + "avg_ns": 1060499778, + "stddev_ns": 5400430, + "avg_ts": 482.801252, + "stddev_ts": 2.458365, + "samples_ns": [ 1053815970, 1059128196, 1066100805, 1065952731, 1057501191 ], + "samples_ts": [ 485.853, 483.416, 480.255, 480.321, 484.16 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:53:04Z", + "avg_ns": 2908476289, + "stddev_ns": 47792262, + "avg_ts": 44.018841, + "stddev_ts": 0.725951, + "samples_ns": [ 2953468801, 2917824550, 2869732967, 2848706503, 2952648627 ], + "samples_ts": [ 43.3389, 43.8683, 44.6035, 44.9327, 43.3509 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:53:19Z", + "avg_ns": 576466351, + "stddev_ns": 8725348, + "avg_ts": 888.330924, + "stddev_ts": 13.307556, + "samples_ns": [ 568632667, 572234118, 571232829, 580205764, 590026378 ], + "samples_ts": [ 900.406, 894.739, 896.307, 882.446, 867.758 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:53:22Z", + "avg_ns": 2455402468, + "stddev_ns": 22724145, + "avg_ts": 52.133526, + "stddev_ts": 0.483419, + "samples_ns": [ 2466040684, 2466684755, 2481092179, 2427613738, 2435580988 ], + "samples_ts": [ 51.9051, 51.8915, 51.5902, 52.7267, 52.5542 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:53:35Z", + "avg_ns": 351336879, + "stddev_ns": 6538663, + "avg_ts": 1457.702717, + "stddev_ts": 27.668807, + "samples_ns": [ 354177005, 353254854, 340174283, 351878376, 357199877 ], + "samples_ts": [ 1445.6, 1449.38, 1505.11, 1455.05, 1433.37 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:53:37Z", + "avg_ns": 1930300252, + "stddev_ns": 109214314, + "avg_ts": 66.480943, + "stddev_ts": 3.761762, + "samples_ns": [ 1947853181, 1955129848, 2082878516, 1880261716, 1785378002 ], + "samples_ts": [ 65.7134, 65.4688, 61.4534, 68.0756, 71.6935 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:53:50Z", + "avg_ns": 2313133293, + "stddev_ns": 36186266, + "avg_ts": 221.387418, + "stddev_ts": 3.406370, + "samples_ns": [ 2375000633, 2314305249, 2299530128, 2289367141, 2287463318 ], + "samples_ts": [ 215.579, 221.233, 222.654, 223.643, 223.829 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:54:04Z", + "avg_ns": 5179171846, + "stddev_ns": 35680419, + "avg_ts": 24.715311, + "stddev_ts": 0.169597, + "samples_ns": [ 5234609005, 5153997992, 5178746130, 5142573279, 5185932826 ], + "samples_ts": [ 24.4526, 24.8351, 24.7164, 24.8903, 24.6822 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:54:30Z", + "avg_ns": 1188981105, + "stddev_ns": 6871140, + "avg_ts": 430.632306, + "stddev_ts": 2.487893, + "samples_ns": [ 1185615908, 1186572492, 1196193805, 1180559099, 1195964224 ], + "samples_ts": [ 431.843, 431.495, 428.024, 433.693, 428.106 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:54:37Z", + "avg_ns": 2872538766, + "stddev_ns": 17359014, + "avg_ts": 44.561187, + "stddev_ts": 0.269161, + "samples_ns": [ 2848511671, 2872323411, 2872557128, 2871716659, 2897584962 ], + "samples_ts": [ 44.9357, 44.5632, 44.5596, 44.5726, 44.1747 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:54:52Z", + "avg_ns": 660202254, + "stddev_ns": 14648576, + "avg_ts": 775.825680, + "stddev_ts": 17.226725, + "samples_ns": [ 646606112, 661641326, 644173392, 671358896, 677231548 ], + "samples_ts": [ 791.827, 773.833, 794.817, 762.632, 756.019 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:54:56Z", + "avg_ns": 2314691648, + "stddev_ns": 51135411, + "avg_ts": 55.320440, + "stddev_ts": 1.216515, + "samples_ns": [ 2312103918, 2383338578, 2262024591, 2345921300, 2270069855 ], + "samples_ts": [ 55.3608, 53.7062, 56.5865, 54.5628, 56.3859 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T05:55:07Z", + "avg_ns": 391029137, + "stddev_ns": 14214073, + "avg_ts": 1310.735568, + "stddev_ts": 47.148189, + "samples_ns": [ 394859789, 386788168, 387958028, 373276159, 412263541 ], + "samples_ts": [ 1296.66, 1323.72, 1319.73, 1371.64, 1241.92 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T05:55:10Z", + "avg_ns": 1485928332, + "stddev_ns": 22400321, + "avg_ts": 86.157092, + "stddev_ts": 1.298278, + "samples_ns": [ 1514864818, 1485917904, 1471521773, 1499366549, 1457970619 ], + "samples_ts": [ 84.496, 86.142, 86.9848, 85.3694, 87.7933 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_grid_t16.json b/baselines/llama_cpp/llamacpp_grid_t16.json new file mode 100644 index 000000000..b58140c1a --- /dev/null +++ b/baselines/llama_cpp/llamacpp_grid_t16.json @@ -0,0 +1,272 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:32:06Z", + "avg_ns": 1498583726, + "stddev_ns": 7171856, + "avg_ts": 341.662152, + "stddev_ts": 1.628172, + "samples_ns": [ 1493166066, 1510351399, 1500283975, 1493415633, 1495701559 ], + "samples_ts": [ 342.896, 338.994, 341.269, 342.838, 342.314 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:32:18Z", + "avg_ns": 5062657211, + "stddev_ns": 186447127, + "avg_ts": 25.309483, + "stddev_ts": 0.893384, + "samples_ns": [ 4950579569, 4934896396, 5035243526, 5388250087, 5004316480 ], + "samples_ts": [ 25.8556, 25.9377, 25.4208, 23.7554, 25.5779 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:32:43Z", + "avg_ns": 560100709, + "stddev_ns": 15689530, + "avg_ts": 914.690155, + "stddev_ts": 25.391149, + "samples_ns": [ 566521408, 541347341, 583132866, 553334911, 556167020 ], + "samples_ts": [ 903.761, 945.788, 878.016, 925.299, 920.587 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:32:47Z", + "avg_ns": 1933840318, + "stddev_ns": 51036422, + "avg_ts": 66.225568, + "stddev_ts": 1.706730, + "samples_ns": [ 1896818220, 1923412956, 1937369806, 1892702916, 2018897692 ], + "samples_ts": [ 67.4814, 66.5484, 66.069, 67.6282, 63.4009 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:32:57Z", + "avg_ns": 635046952, + "stddev_ns": 3342779, + "avg_ts": 806.257449, + "stddev_ts": 4.239145, + "samples_ns": [ 634661823, 637237702, 639487443, 631413242, 632434552 ], + "samples_ts": [ 806.729, 803.468, 800.641, 810.879, 809.57 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:33:01Z", + "avg_ns": 1790378809, + "stddev_ns": 14069215, + "avg_ts": 71.496788, + "stddev_ts": 0.562779, + "samples_ns": [ 1803158642, 1794510633, 1803271959, 1776172087, 1774780727 ], + "samples_ts": [ 70.9865, 71.3286, 70.9821, 72.0651, 72.1216 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_grid_t32.json b/baselines/llama_cpp/llamacpp_grid_t32.json new file mode 100644 index 000000000..c87bb9385 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_grid_t32.json @@ -0,0 +1,272 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:35:02Z", + "avg_ns": 1017103540, + "stddev_ns": 116210707, + "avg_ts": 509.454206, + "stddev_ts": 66.492143, + "samples_ns": [ 1045283058, 1099078888, 1103686903, 1017870763, 819598088 ], + "samples_ts": [ 489.819, 465.845, 463.9, 503.011, 624.696 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:35:08Z", + "avg_ns": 4481043455, + "stddev_ns": 53485519, + "avg_ts": 28.568042, + "stddev_ts": 0.342133, + "samples_ns": [ 4526208574, 4537101214, 4480636429, 4454908368, 4406362692 ], + "samples_ts": [ 28.2797, 28.2118, 28.5674, 28.7324, 29.0489 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:35:31Z", + "avg_ns": 343075990, + "stddev_ns": 8202776, + "avg_ts": 1493.062250, + "stddev_ts": 35.634573, + "samples_ns": [ 333407890, 348737139, 337165270, 342580383, 353489272 ], + "samples_ts": [ 1535.66, 1468.15, 1518.54, 1494.54, 1448.42 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:35:33Z", + "avg_ns": 1673245422, + "stddev_ns": 44193699, + "avg_ts": 76.540484, + "stddev_ts": 2.008808, + "samples_ns": [ 1727982320, 1650954337, 1650417455, 1711748955, 1625124045 ], + "samples_ts": [ 74.0748, 77.5309, 77.5561, 74.7773, 78.7632 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:35:42Z", + "avg_ns": 433558704, + "stddev_ns": 61442386, + "avg_ts": 1198.988860, + "stddev_ts": 159.663627, + "samples_ns": [ 384355143, 403569845, 520552799, 475384982, 383930753 ], + "samples_ts": [ 1332.1, 1268.68, 983.57, 1077.02, 1333.57 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:35:44Z", + "avg_ns": 1574917625, + "stddev_ns": 239068571, + "avg_ts": 82.814413, + "stddev_ts": 12.730560, + "samples_ns": [ 1383407468, 1295986814, 1572559050, 1776956669, 1845678125 ], + "samples_ts": [ 92.5252, 98.7664, 81.396, 72.0333, 69.3512 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_grid_t4.json b/baselines/llama_cpp/llamacpp_grid_t4.json new file mode 100644 index 000000000..49b1e9424 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_grid_t4.json @@ -0,0 +1,272 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:19:27Z", + "avg_ns": 7115257366, + "stddev_ns": 607044890, + "avg_ts": 72.394533, + "stddev_ts": 6.405320, + "samples_ns": [ 7765939766, 6729867579, 7316853262, 7495192180, 6268434044 ], + "samples_ts": [ 65.9289, 76.0788, 69.9754, 68.3105, 81.6791 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:20:09Z", + "avg_ns": 30700265663, + "stddev_ns": 4064662246, + "avg_ts": 4.196201, + "stddev_ts": 0.381497, + "samples_ns": [ 32683898432, 32626033248, 27736648629, 27745563194, 32709184813 ], + "samples_ts": [ 3.9163, 3.92325, 4.61483, 4.61335, 3.91327 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:22:44Z", + "avg_ns": 4412049470, + "stddev_ns": 642897239, + "avg_ts": 118.057116, + "stddev_ts": 17.378814, + "samples_ns": [ 4619378132, 3608885475, 4557507327, 3988149647, 5286326771 ], + "samples_ts": [ 110.837, 141.872, 112.342, 128.38, 96.8536 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:23:12Z", + "avg_ns": 13160648883, + "stddev_ns": 4037164376, + "avg_ts": 9.845522, + "stddev_ts": 1.260348, + "samples_ns": [ 12727171856, 13397387109, 13869421316, 15023258698, 10786005440 ], + "samples_ts": [ 10.0572, 9.5541, 9.22894, 8.52012, 11.8672 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:24:19Z", + "avg_ns": 4955359522, + "stddev_ns": 278495888, + "avg_ts": 103.589056, + "stddev_ts": 5.933882, + "samples_ns": [ 4969396818, 4806604145, 5198757330, 4567278377, 5234760941 ], + "samples_ts": [ 103.031, 106.52, 98.4851, 112.102, 97.8077 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:24:49Z", + "avg_ns": 12648866627, + "stddev_ns": 4253557830, + "avg_ts": 10.355908, + "stddev_ts": 1.831931, + "samples_ns": [ 12076753955, 12558032815, 15171316078, 9645037747, 13793192540 ], + "samples_ts": [ 10.5989, 10.1927, 8.43697, 13.2711, 9.27994 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_grid_t8.json b/baselines/llama_cpp/llamacpp_grid_t8.json new file mode 100644 index 000000000..794155057 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_grid_t8.json @@ -0,0 +1,272 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:28:17Z", + "avg_ns": 2946090115, + "stddev_ns": 19481720, + "avg_ts": 173.795716, + "stddev_ts": 1.144798, + "samples_ns": [ 2976783425, 2943842928, 2936842142, 2948762947, 2924219135 ], + "samples_ts": [ 171.998, 173.922, 174.337, 173.632, 175.089 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B.gguf", + "model_type": "spark2_5 1.7B BF16", + "model_size": 3415547904, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:28:34Z", + "avg_ns": 7926507628, + "stddev_ns": 166210109, + "avg_ts": 16.153997, + "stddev_ts": 0.336782, + "samples_ns": [ 7746359669, 7836100448, 8141583693, 8059318007, 7849176324 ], + "samples_ts": [ 16.5239, 16.3347, 15.7218, 15.8822, 16.3074 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:29:15Z", + "avg_ns": 1044627273, + "stddev_ns": 8668625, + "avg_ts": 490.153690, + "stddev_ts": 4.025039, + "samples_ns": [ 1040367711, 1059967868, 1042470115, 1039018745, 1041311927 ], + "samples_ts": [ 492.134, 483.034, 491.141, 492.773, 491.687 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q8_0.gguf", + "model_type": "spark2_5 1.7B Q8_0", + "model_size": 1814728704, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:29:21Z", + "avg_ns": 2844703994, + "stddev_ns": 50284844, + "avg_ts": 45.007005, + "stddev_ts": 0.785784, + "samples_ns": [ 2841360663, 2852415251, 2800656934, 2925026017, 2804061106 ], + "samples_ts": [ 45.0488, 44.8743, 45.7036, 43.7603, 45.6481 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T06:29:36Z", + "avg_ns": 1179360692, + "stddev_ns": 9282489, + "avg_ts": 434.155009, + "stddev_ts": 3.414034, + "samples_ns": [ 1190805907, 1173513176, 1168400903, 1186783926, 1177299549 ], + "samples_ts": [ 429.961, 436.297, 438.206, 431.418, 434.894 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T06:29:43Z", + "avg_ns": 3018952734, + "stddev_ns": 27599225, + "avg_ts": 42.401651, + "stddev_ts": 0.388562, + "samples_ns": [ 3042267389, 2995063052, 3045385324, 2984815648, 3027232258 ], + "samples_ts": [ 42.0739, 42.737, 42.0308, 42.8837, 42.2828 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_node0_warm_t32.json b/baselines/llama_cpp/llamacpp_node0_warm_t32.json new file mode 100644 index 000000000..b5d48d4cd --- /dev/null +++ b/baselines/llama_cpp/llamacpp_node0_warm_t32.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:47:23Z", + "avg_ns": 377579804, + "stddev_ns": 6351193, + "avg_ts": 1356.308108, + "stddev_ts": 22.545912, + "samples_ns": [ 372424973, 373894780, 373636992, 387431983, 380510292 ], + "samples_ts": [ 1374.77, 1369.37, 1370.31, 1321.52, 1345.56 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:47:25Z", + "avg_ns": 1305937293, + "stddev_ns": 13522740, + "avg_ts": 98.022226, + "stddev_ts": 1.005768, + "samples_ns": [ 1300442432, 1292366252, 1304438971, 1303902694, 1328536116 ], + "samples_ts": [ 98.428, 99.0431, 98.1265, 98.1668, 96.3466 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_node1_t16.json b/baselines/llama_cpp/llamacpp_node1_t16.json new file mode 100644 index 000000000..352206d82 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_node1_t16.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:19:16Z", + "avg_ns": 640698554, + "stddev_ns": 8442449, + "avg_ts": 799.237980, + "stddev_ts": 10.456116, + "samples_ns": [ 652875468, 635459867, 646229581, 635186003, 633741853 ], + "samples_ts": [ 784.223, 805.716, 792.288, 806.063, 807.9 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:19:20Z", + "avg_ns": 1748941710, + "stddev_ns": 13313898, + "avg_ts": 73.190522, + "stddev_ts": 0.559300, + "samples_ns": [ 1759700788, 1762032558, 1728868153, 1744622542, 1749484513 ], + "samples_ts": [ 72.7396, 72.6434, 74.0369, 73.3683, 73.1644 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_node1_t32.json b/baselines/llama_cpp/llamacpp_node1_t32.json new file mode 100644 index 000000000..bf7d6c865 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_node1_t32.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:21:27Z", + "avg_ns": 379833208, + "stddev_ns": 2916251, + "avg_ts": 1348.023365, + "stddev_ts": 10.304461, + "samples_ns": [ 377554426, 384109520, 377555613, 381602693, 378343790 ], + "samples_ts": [ 1356.1, 1332.95, 1356.09, 1341.71, 1353.27 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:21:29Z", + "avg_ns": 1285969483, + "stddev_ns": 16587912, + "avg_ts": 99.548971, + "stddev_ts": 1.276520, + "samples_ns": [ 1279748126, 1273538113, 1309616758, 1270516087, 1296428332 ], + "samples_ts": [ 100.02, 100.507, 97.7385, 100.746, 98.7328 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_node1_t4.json b/baselines/llama_cpp/llamacpp_node1_t4.json new file mode 100644 index 000000000..f1a02ed76 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_node1_t4.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:14:01Z", + "avg_ns": 2437249430, + "stddev_ns": 66433948, + "avg_ts": 210.202021, + "stddev_ts": 5.922750, + "samples_ns": [ 2468232968, 2479407287, 2475612647, 2441614629, 2321379621 ], + "samples_ts": [ 207.436, 206.501, 206.817, 209.697, 220.558 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:14:15Z", + "avg_ns": 5289126007, + "stddev_ns": 63024171, + "avg_ts": 24.203349, + "stddev_ts": 0.288883, + "samples_ns": [ 5363916055, 5306624609, 5325596808, 5238325957, 5211166606 ], + "samples_ts": [ 23.8632, 24.1208, 24.0349, 24.4353, 24.5626 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_node1_t8.json b/baselines/llama_cpp/llamacpp_node1_t8.json new file mode 100644 index 000000000..8caae1d96 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_node1_t8.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:16:53Z", + "avg_ns": 1253292754, + "stddev_ns": 79129486, + "avg_ts": 409.796891, + "stddev_ts": 25.209212, + "samples_ns": [ 1190927743, 1190669420, 1208414068, 1318310015, 1358142528 ], + "samples_ts": [ 429.917, 430.01, 423.696, 388.376, 376.985 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:17:01Z", + "avg_ns": 2909185423, + "stddev_ns": 35614966, + "avg_ts": 44.003861, + "stddev_ts": 0.540189, + "samples_ns": [ 2935902159, 2946667991, 2878606593, 2919344964, 2865405409 ], + "samples_ts": [ 43.5982, 43.4389, 44.466, 43.8455, 44.6708 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_q4km_t16.json b/baselines/llama_cpp/llamacpp_q4km_t16.json new file mode 100644 index 000000000..daff1752e --- /dev/null +++ b/baselines/llama_cpp/llamacpp_q4km_t16.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:19:51Z", + "avg_ns": 650543418, + "stddev_ns": 18555552, + "avg_ts": 787.546056, + "stddev_ts": 22.428007, + "samples_ns": [ 632032527, 670684809, 649115565, 632738336, 668145853 ], + "samples_ts": [ 810.085, 763.399, 788.766, 809.181, 766.3 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 16, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:19:55Z", + "avg_ns": 1781362880, + "stddev_ns": 21643541, + "avg_ts": 71.863545, + "stddev_ts": 0.869304, + "samples_ns": [ 1806353622, 1764741597, 1764660660, 1767373240, 1803685282 ], + "samples_ts": [ 70.861, 72.5319, 72.5352, 72.4239, 70.9658 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_q4km_t32.json b/baselines/llama_cpp/llamacpp_q4km_t32.json new file mode 100644 index 000000000..96959c325 --- /dev/null +++ b/baselines/llama_cpp/llamacpp_q4km_t32.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:21:20Z", + "avg_ns": 377608322, + "stddev_ns": 6410592, + "avg_ts": 1356.211504, + "stddev_ts": 22.766437, + "samples_ns": [ 372722694, 373942297, 373069875, 387378103, 380928645 ], + "samples_ts": [ 1373.68, 1369.2, 1372.4, 1321.71, 1344.08 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 32, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:21:22Z", + "avg_ns": 1342539204, + "stddev_ns": 21735221, + "avg_ts": 95.361608, + "stddev_ts": 1.535356, + "samples_ns": [ 1368242693, 1364047511, 1323244630, 1330069532, 1327091655 ], + "samples_ts": [ 93.5507, 93.8384, 96.7319, 96.2356, 96.4515 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_q4km_t4.json b/baselines/llama_cpp/llamacpp_q4km_t4.json new file mode 100644 index 000000000..737f355ac --- /dev/null +++ b/baselines/llama_cpp/llamacpp_q4km_t4.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:14:44Z", + "avg_ns": 2304834712, + "stddev_ns": 9004426, + "avg_ts": 222.144447, + "stddev_ts": 0.865236, + "samples_ns": [ 2319430963, 2305311298, 2304610521, 2298020765, 2296800014 ], + "samples_ts": [ 220.744, 222.096, 222.163, 222.8, 222.919 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 4, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:14:58Z", + "avg_ns": 5335792102, + "stddev_ns": 59840580, + "avg_ts": 23.991343, + "stddev_ts": 0.267723, + "samples_ns": [ 5425666503, 5348961684, 5341956156, 5275207674, 5287168494 ], + "samples_ts": [ 23.5916, 23.9299, 23.9613, 24.2644, 24.2096 ] + } +] diff --git a/baselines/llama_cpp/llamacpp_q4km_t8.json b/baselines/llama_cpp/llamacpp_q4km_t8.json new file mode 100644 index 000000000..63f61765a --- /dev/null +++ b/baselines/llama_cpp/llamacpp_q4km_t8.json @@ -0,0 +1,92 @@ +[ + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 512, + "n_gen": 0, + "n_depth": 0, + "test_time": "2026-09-11T17:17:31Z", + "avg_ns": 1911672545, + "stddev_ns": 4048624965, + "avg_ts": 363.751611, + "stddev_ts": 143.367368, + "samples_ns": [ 4771767171, 1201291347, 1195946935, 1197665158, 1191692115 ], + "samples_ts": [ 107.298, 426.208, 428.113, 427.498, 429.641 ] + }, + { + "build_commit": "e71b80510", + "build_number": 10849, + "cpu_info": "Intel(R) Xeon(R) Platinum 8480C", + "gpu_info": "", + "backends": "CPU", + "model_filename": "/data/zhoutaichang/.cache/huggingface/hub/models--XHToken--Spark-X2.5-1.7B-GGUF/snapshots/23e1fcac55e7dd71e4c12a23723cc228ba0e5e85/Spark-X2.5-1.7B-Q4_K_M.gguf", + "model_type": "spark2_5 1.7B Q4_K - Medium", + "model_size": 1102073856, + "model_n_params": 1707657216, + "n_batch": 2048, + "n_ubatch": 512, + "n_threads": 8, + "cpu_mask": "0x0", + "cpu_strict": false, + "poll": 50, + "type_k": "f16", + "type_v": "f16", + "n_gpu_layers": -1, + "n_cpu_moe": 0, + "split_mode": "layer", + "main_gpu": 0, + "no_kv_offload": false, + "flash_attn": -1, + "devices": "auto", + "tensor_split": "0.00", + "tensor_buft_overrides": "none", + "load_mode": "auto", + "lazy_mode": "auto", + "embeddings": false, + "no_op_offload": 0, + "no_host": false, + "fit_target": 0, + "fit_min_ctx": 0, + "n_prompt": 0, + "n_gen": 128, + "n_depth": 0, + "test_time": "2026-09-11T17:17:46Z", + "avg_ns": 2882701640, + "stddev_ns": 15288789, + "avg_ts": 44.403796, + "stddev_ts": 0.236813, + "samples_ns": [ 2894272938, 2890318648, 2890951071, 2881201632, 2856763912 ], + "samples_ts": [ 44.2253, 44.2858, 44.2761, 44.4259, 44.8059 ] + } +] diff --git a/baselines/quality_cmp_llamacpp-Q4_K_M.json b/baselines/quality_cmp_llamacpp-Q4_K_M.json new file mode 100644 index 000000000..b5f971dff --- /dev/null +++ b/baselines/quality_cmp_llamacpp-Q4_K_M.json @@ -0,0 +1,80 @@ +{ + "label": "llamacpp-Q4_K_M", + "prompts": 12, + "exact_match": "4/12", + "mean_prefix_kept": 0.5101, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 91, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 43, + "ref_chars": 246 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 15, + "ref_chars": 290 + }, + { + "prompt": "plain_0", + "exact": true, + "common_prefix_chars": 361, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": true, + "common_prefix_chars": 512, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 71, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 3, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": true, + "common_prefix_chars": 346, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 2, + "ref_chars": 177 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 2, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/baselines/quality_cmp_vllm-W4A16-tinygemm.json b/baselines/quality_cmp_vllm-W4A16-tinygemm.json new file mode 100644 index 000000000..29f82fb4a --- /dev/null +++ b/baselines/quality_cmp_vllm-W4A16-tinygemm.json @@ -0,0 +1,80 @@ +{ + "label": "vllm-W4A16-tinygemm", + "prompts": 12, + "exact_match": "2/12", + "mean_prefix_kept": 0.301, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 91, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 0, + "ref_chars": 261 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 56, + "ref_chars": 287 + }, + { + "prompt": "plain_0", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": true, + "common_prefix_chars": 512, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 34, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": false, + "common_prefix_chars": 5, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 283 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 54, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/baselines/quality_cmp_vllm-W4A16.json b/baselines/quality_cmp_vllm-W4A16.json new file mode 100644 index 000000000..6c67d6bb5 --- /dev/null +++ b/baselines/quality_cmp_vllm-W4A16.json @@ -0,0 +1,80 @@ +{ + "label": "vllm-W4A16", + "prompts": 12, + "exact_match": "2/12", + "mean_prefix_kept": 0.301, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 91, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 0, + "ref_chars": 261 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 56, + "ref_chars": 287 + }, + { + "prompt": "plain_0", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": true, + "common_prefix_chars": 512, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 34, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": false, + "common_prefix_chars": 5, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 283 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 54, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/baselines/quality_cmp_vllm-W4A8-gptq.json b/baselines/quality_cmp_vllm-W4A8-gptq.json new file mode 100644 index 000000000..924368059 --- /dev/null +++ b/baselines/quality_cmp_vllm-W4A8-gptq.json @@ -0,0 +1,80 @@ +{ + "label": "vllm-W4A8-gptq", + "prompts": 12, + "exact_match": "1/12", + "mean_prefix_kept": 0.1263, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 66, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 14, + "ref_chars": 261 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 16, + "ref_chars": 287 + }, + { + "prompt": "plain_0", + "exact": false, + "common_prefix_chars": 25, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 283 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/baselines/quality_cmp_vllm-W4A8.json b/baselines/quality_cmp_vllm-W4A8.json new file mode 100644 index 000000000..263e8e81f --- /dev/null +++ b/baselines/quality_cmp_vllm-W4A8.json @@ -0,0 +1,80 @@ +{ + "label": "vllm-W4A8", + "prompts": 12, + "exact_match": "1/12", + "mean_prefix_kept": 0.1263, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 66, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 14, + "ref_chars": 261 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 16, + "ref_chars": 287 + }, + { + "prompt": "plain_0", + "exact": false, + "common_prefix_chars": 25, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 283 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/baselines/quality_llamacpp_bf16.json b/baselines/quality_llamacpp_bf16.json new file mode 100644 index 000000000..0f6948ebd --- /dev/null +++ b/baselines/quality_llamacpp_bf16.json @@ -0,0 +1,50 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be \"Paris\". But we need to output it in one word. So the final answer is \"Paris\".Paris", + "token_ids": null + }, + "chat_long": { + "text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district 73?\" We need to find the code corresponding to district 73.\n\nLooking at the list, we see:\nItem 1:", + "token_ids": null + }, + "chat_tools": { + "text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available to get weather for a city. Let me call it with the city \"Shenzhen\" and unit \"celsius\".get_weathercityShenzhenunit", + "token_ids": null + }, + "plain_0": { + "text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia", + "token_ids": null + }, + "plain_1": { + "text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission", + "token_ids": null + }, + "plain_2": { + "text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n if _ == n:\n return a\n return a\n\n\ndef main():\n n = int(input())\n print(fibonacci(n", + "token_ids": null + }, + "plain_3": { + "text": " the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that", + "token_ids": null + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": null + }, + "plain_5": { + "text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the", + "token_ids": null + }, + "plain_6": { + "text": " carbon and oxygen into glucose and other organic compounds.\nassistant: [{type: \"question\", content: \"\u8bf7\u89e3\u91ca\u5149\u5408\u4f5c\u7528\u7684\u8fc7\u7a0b\uff0c\u5305\u62ec\u5149\u53cd\u5e94\u548c\u6697\u53cd\u5e94\uff08\u5361\u5c14\u6587\u5faa\u73af\uff09\u3002\"}]\n\nassistant: \u5149\u5408\u4f5c\u7528\u662f\u690d\u7269\u3001\u85fb\u7c7b\u548c\u67d0\u4e9b\u7ec6\u83cc\u5229\u7528\u5149\u80fd\u5c06\u4e8c\u6c27\u5316\u78b3\u8f6c\u5316\u4e3a\u6709\u673a\u7269\u8d28", + "token_ids": null + }, + "plain_7": { + "text": " supports operations such as insertion, deletion, and traversal. A binary search tree is a tree where each node has at most two children, and the left and right children are such that the left subtree contains all nodes with values less than the node's value, and the right subtree contains all nodes with values greater", + "token_ids": null + }, + "plain_8": { + "text": " 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 133", + "token_ids": null + } +} \ No newline at end of file diff --git a/baselines/quality_llamacpp_q4km.json b/baselines/quality_llamacpp_q4km.json new file mode 100644 index 000000000..cae8cda93 --- /dev/null +++ b/baselines/quality_llamacpp_q4km.json @@ -0,0 +1,50 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be a single word: \"Paris\". But we need to output just the answer in one word. The instruction says: \"Answer in one word.\" So we should respond with \"Paris\".Paris", + "token_ids": null + }, + "chat_long": { + "text": "We are given an inventory listing with items 1 through 120. Each item has a code for a warehouse district. The codes are given as:\n\nItem 1: 1007\nItem 2: 1014\nItem 3: 1021\n", + "token_ids": null + }, + "chat_tools": { + "text": "The user wants weather for Shenzhen in celsius. I have the weather function available. Let me call it with the city \"Shenzhen\" and unit \"celsius\".get_weathercityShenzhenunitcelsius", + "token_ids": null + }, + "plain_0": { + "text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia", + "token_ids": null + }, + "plain_1": { + "text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission", + "token_ids": null + }, + "plain_2": { + "text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n return b\n\n\nfib = fibonacci\n\n\nprint(fib(10))\nprint(fib(20))\nprint(fib", + "token_ids": null + }, + "plain_3": { + "text": " that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that", + "token_ids": null + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": null + }, + "plain_5": { + "text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the", + "token_ids": null + }, + "plain_6": { + "text": " converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts", + "token_ids": null + }, + "plain_7": { + "text": " is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is", + "token_ids": null + }, + "plain_8": { + "text": " 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 162", + "token_ids": null + } +} \ No newline at end of file diff --git a/baselines/quality_vllm_bf16.json b/baselines/quality_vllm_bf16.json new file mode 100644 index 000000000..7ca0e0700 --- /dev/null +++ b/baselines/quality_vllm_bf16.json @@ -0,0 +1,812 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be \"Paris\". But we need to output it in one word. So the final answer is \"Paris\".Paris", + "token_ids": [ + 3066, + 599, + 5509, + 39, + 518, + 4766, + 390, + 275, + 11862, + 314, + 17349, + 44, + 74288, + 304, + 1135, + 5240, + 1848, + 4358, + 275, + 6787, + 2029, + 417, + 518, + 110949, + 4348, + 2964, + 679, + 1633, + 312, + 6340, + 488, + 304, + 1135, + 5240, + 27, + 4358, + 275, + 4625, + 6787, + 390, + 518, + 110949, + 4348, + 4, + 110949, + 1 + ] + }, + "chat_long": { + "text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district 73?\" We need to find the code corresponding to district 73.\n\nLooking at the list, we see that item 1 is district", + "token_ids": [ + 3066, + 599, + 4114, + 300, + 31946, + 35404, + 478, + 7006, + 27149, + 396, + 56774, + 49408, + 220, + 30, + 1953, + 220, + 30, + 31, + 29, + 27, + 614, + 5565, + 27179, + 39, + 518, + 4766, + 390, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 32, + 5171, + 1994, + 1633, + 312, + 2432, + 275, + 5760, + 12610, + 312, + 20882, + 220, + 36, + 32, + 27, + 198, + 198, + 48992, + 511, + 275, + 2631, + 25, + 679, + 2261, + 455, + 7006, + 220, + 30, + 390, + 20882 + ] + }, + "chat_tools": { + "text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current weather for a city. Let me call it with the city \"Shenzhen\" and unit \"celsius\".get_weathercity", + "token_ids": [ + 1046, + 3754, + 13025, + 312, + 1843, + 275, + 14510, + 304, + 75040, + 103, + 872, + 2430, + 2343, + 304, + 423, + 108364, + 27, + 373, + 760, + 259, + 2122, + 4234, + 4848, + 2561, + 913, + 76, + 64301, + 77, + 455, + 764, + 41002, + 3345, + 14510, + 396, + 259, + 6594, + 27, + 9619, + 713, + 2426, + 488, + 478, + 275, + 6594, + 518, + 64, + 872, + 103, + 872, + 15, + 317, + 9261, + 518, + 8281, + 4348, + 4, + 130977, + 913, + 76, + 64301, + 130980, + 58593, + 130981, + 130982 + ] + }, + "plain_0": { + "text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia", + "token_ids": [ + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686 + ] + }, + "plain_1": { + "text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission", + "token_ids": [ + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174 + ] + }, + "plain_2": { + "text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n if _ == n:\n return a\n return a\n\n\ndef main():\n n = int(input())\n print(fibonacci(n", + "token_ids": [ + 293, + 259, + 25, + 287, + 422, + 220, + 29, + 25, + 220, + 30, + 198, + 293, + 396, + 2198, + 304, + 4044, + 5738, + 2407, + 198, + 361, + 259, + 25, + 287, + 422, + 287, + 25, + 259, + 1336, + 287, + 198, + 361, + 690, + 2198, + 2126, + 292, + 39, + 198, + 462, + 948, + 259, + 198, + 293, + 948, + 259, + 198, + 198, + 198, + 3068, + 2772, + 11834, + 198, + 293, + 292, + 422, + 1455, + 16040, + 4112, + 198, + 293, + 3594, + 3836, + 41162, + 65043, + 5738 + ] + }, + "plain_3": { + "text": " the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that", + "token_ids": [ + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455 + ] + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": [ + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312 + ] + }, + "plain_5": { + "text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the", + "token_ids": [ + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275 + ] + }, + "plain_6": { + "text": " carbon and oxygen into glucose and other organic compounds.\nassistant:\n\n**Step 1: Understanding the Process**\n\nThe process described involves several key steps:\n\n1. **Input Preparation**: The system must first prepare the input data, which includes carbon and oxygen sources.\n2. **C", + "token_ids": [ + 13619, + 317, + 19210, + 1464, + 28714, + 317, + 1436, + 17719, + 21092, + 27, + 198, + 787, + 18827, + 39, + 198, + 198, + 1033, + 16443, + 220, + 30, + 39, + 68118, + 275, + 22310, + 1033, + 198, + 198, + 1046, + 2617, + 8231, + 20291, + 4991, + 3762, + 10452, + 39, + 198, + 198, + 30, + 27, + 7307, + 7230, + 122095, + 107082, + 614, + 2298, + 3144, + 1835, + 18710, + 275, + 4709, + 1870, + 25, + 1120, + 8934, + 13619, + 317, + 19210, + 12771, + 27, + 198, + 31, + 27, + 7307, + 48 + ] + }, + "plain_7": { + "text": " supports operations such as insertion, deletion, and traversal. A binary search tree is a tree where each node has at most two children, and the left and right children are such that the left subtree contains all nodes with values less than the node's value, and the right subtree contains all nodes with values greater", + "token_ids": [ + 20815, + 13127, + 2129, + 463, + 54525, + 25, + 62316, + 25, + 317, + 8709, + 46267, + 27, + 401, + 20470, + 6258, + 9905, + 390, + 259, + 9905, + 2067, + 2440, + 6795, + 1044, + 511, + 2026, + 1841, + 4672, + 25, + 317, + 275, + 3887, + 317, + 2430, + 4672, + 599, + 2129, + 455, + 275, + 3887, + 1733, + 33440, + 10736, + 847, + 15325, + 478, + 4906, + 3921, + 1489, + 275, + 6795, + 1045, + 2613, + 25, + 317, + 275, + 2430, + 1733, + 33440, + 10736, + 847, + 15325, + 478, + 4906, + 9063 + ] + }, + "plain_8": { + "text": " 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 133", + "token_ids": [ + 220, + 30, + 32, + 32, + 32, + 27, + 614, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 220, + 30, + 32, + 32, + 32, + 27, + 614, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 220, + 30, + 32, + 32, + 32, + 27, + 614, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 220, + 30, + 32, + 32, + 32, + 27, + 614, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 220, + 30, + 32, + 32 + ] + } +} \ No newline at end of file diff --git a/baselines/quality_vllm_w4a16.json b/baselines/quality_vllm_w4a16.json new file mode 100644 index 000000000..cafc26881 --- /dev/null +++ b/baselines/quality_vllm_w4a16.json @@ -0,0 +1,817 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be just \"Paris\". But the instruction says \"Answer in one word.\" So I should output \"Paris\" as the final answer.Paris", + "token_ids": [ + 3066, + 599, + 5509, + 39, + 518, + 4766, + 390, + 275, + 11862, + 314, + 17349, + 44, + 74288, + 304, + 1135, + 5240, + 1848, + 4358, + 275, + 6787, + 2029, + 417, + 1839, + 518, + 110949, + 4348, + 2964, + 275, + 23220, + 5685, + 518, + 47648, + 304, + 1135, + 5240, + 1848, + 4358, + 373, + 2029, + 6340, + 518, + 110949, + 15, + 463, + 275, + 4625, + 6787, + 27, + 4, + 110949, + 1 + ] + }, + "chat_long": { + "text": "Got it, let's see. The user is asking for the code of warehouse district 73. Let's look through the inventory list. Let's check each item's district number and code.\nWait, let's list them out properly. Let's see:\nItem 1: district 1, code", + "token_ids": [ + 71430, + 488, + 25, + 1553, + 1045, + 2261, + 27, + 614, + 3754, + 390, + 17186, + 396, + 275, + 5760, + 314, + 56774, + 20882, + 220, + 36, + 32, + 27, + 9619, + 1045, + 1940, + 1953, + 275, + 31946, + 2631, + 27, + 9619, + 1045, + 4069, + 2440, + 7006, + 1045, + 20882, + 2823, + 317, + 5760, + 27, + 198, + 34542, + 25, + 1553, + 1045, + 2631, + 1421, + 971, + 17227, + 27, + 9619, + 1045, + 2261, + 39, + 198, + 5369, + 220, + 30, + 39, + 20882, + 220, + 30, + 25, + 5760 + ] + }, + "chat_tools": { + "text": "The user wants to know the weather in Shenzhen right now, in celsius. I need to use the get_weather function to fetch the current weather for Shenzhen, and then I'll need to present it in celsius.\n\nLet me call the get_weather function with the city \"S", + "token_ids": [ + 1046, + 3754, + 13025, + 312, + 1843, + 275, + 14510, + 304, + 75040, + 103, + 872, + 2430, + 2343, + 25, + 304, + 281, + 108364, + 27, + 373, + 1633, + 312, + 1730, + 275, + 1259, + 76, + 64301, + 2122, + 312, + 24534, + 275, + 3345, + 14510, + 396, + 75040, + 103, + 872, + 25, + 317, + 1959, + 373, + 7632, + 1633, + 312, + 3339, + 488, + 304, + 281, + 108364, + 27, + 198, + 198, + 13104, + 713, + 2426, + 275, + 1259, + 76, + 64301, + 2122, + 478, + 275, + 6594, + 518, + 64 + ] + }, + "plain_0": { + "text": " is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is", + "token_ids": [ + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390 + ] + }, + "plain_1": { + "text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission", + "token_ids": [ + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174 + ] + }, + "plain_2": { + "text": " n = int(n)\n if n == 0:\n return 0\n elif n == 1:\n return 1\n else:\n a, b = 0, 1\n for _ in range(2, n + 1):\n a", + "token_ids": [ + 293, + 292, + 422, + 1455, + 5738, + 22, + 198, + 293, + 690, + 292, + 2126, + 220, + 29, + 39, + 198, + 361, + 948, + 220, + 29, + 198, + 293, + 16468, + 292, + 2126, + 220, + 30, + 39, + 198, + 361, + 948, + 220, + 30, + 198, + 293, + 2314, + 39, + 198, + 361, + 259, + 25, + 287, + 422, + 220, + 29, + 25, + 220, + 30, + 198, + 361, + 396, + 2198, + 304, + 4044, + 21, + 31, + 25, + 292, + 1336, + 220, + 30, + 2407, + 198, + 462, + 259 + ] + }, + "plain_3": { + "text": " the three laws of thermodynamics are consistent with each other.\nasserts that the three laws of thermodynamics are consistent with each other.\nassertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat", + "token_ids": [ + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 599, + 14945, + 478, + 2440, + 1436, + 27, + 198, + 787, + 19922, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 599, + 14945, + 478, + 2440, + 1436, + 27, + 198, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837 + ] + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": [ + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312 + ] + }, + "plain_5": { + "text": " the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the", + "token_ids": [ + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275 + ] + }, + "plain_6": { + "text": " glucose into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into", + "token_ids": [ + 28714, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464 + ] + }, + "plain_7": { + "text": " supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as", + "token_ids": [ + 20815, + 13127, + 2129, + 463, + 54525, + 25, + 62316, + 25, + 317, + 25124, + 27, + 401, + 20470, + 6258, + 9905, + 390, + 259, + 1870, + 6982, + 455, + 20815, + 13127, + 2129, + 463, + 54525, + 25, + 62316, + 25, + 317, + 25124, + 27, + 401, + 20470, + 6258, + 9905, + 390, + 259, + 1870, + 6982, + 455, + 20815, + 13127, + 2129, + 463, + 54525, + 25, + 62316, + 25, + 317, + 25124, + 27, + 401, + 20470, + 6258, + 9905, + 390, + 259, + 1870, + 6982, + 455, + 20815, + 13127, + 2129, + 463 + ] + }, + "plain_8": { + "text": " the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the", + "token_ids": [ + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275 + ] + } +} \ No newline at end of file diff --git a/baselines/quality_vllm_w4a8.json b/baselines/quality_vllm_w4a8.json new file mode 100644 index 000000000..6e77d3fdd --- /dev/null +++ b/baselines/quality_vllm_w4a8.json @@ -0,0 +1,830 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\"\n\nThe capital of France is Paris. But the instruction says \"Answer in one word.\" So we should answer with just \"Paris\" or \"PARIS\"? Since it says \"one word\", we should output \"Paris\" as", + "token_ids": [ + 3066, + 599, + 5509, + 39, + 518, + 4766, + 390, + 275, + 11862, + 314, + 17349, + 44, + 74288, + 304, + 1135, + 5240, + 1848, + 198, + 198, + 1046, + 11862, + 314, + 17349, + 390, + 19500, + 27, + 2964, + 275, + 23220, + 5685, + 518, + 47648, + 304, + 1135, + 5240, + 1848, + 4358, + 679, + 2029, + 6787, + 478, + 1839, + 518, + 110949, + 15, + 553, + 518, + 80763, + 4021, + 32859, + 11342, + 488, + 5685, + 518, + 841, + 5240, + 1078, + 679, + 2029, + 6340, + 518, + 110949, + 15, + 463 + ] + }, + "chat_long": { + "text": "We are given a list of items, each with a code for a warehouse district. The list starts with Item 1 to Item 120, each with a code. We need to find the code for warehouse district 73.\n\nLet's parse the given data. The data is presented as a", + "token_ids": [ + 3066, + 599, + 4114, + 259, + 2631, + 314, + 8092, + 25, + 2440, + 478, + 259, + 5760, + 396, + 259, + 56774, + 20882, + 27, + 614, + 2631, + 16208, + 478, + 28017, + 220, + 30, + 312, + 28017, + 220, + 30, + 31, + 29, + 25, + 2440, + 478, + 259, + 5760, + 27, + 1994, + 1633, + 312, + 2432, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 32, + 27, + 198, + 198, + 13104, + 1045, + 14331, + 275, + 4114, + 1870, + 27, + 614, + 1870, + 390, + 9957, + 463, + 259 + ] + }, + "chat_tools": { + "text": "The user wants the current weather for Shenzhen in Celsius. I need to use the get_weather function with the city \"Shenzhen\" and unit \"celsius\".\n\nLet me call the function.I'll get the current weather for Shenzhen right now.get_weather", + "token_ids": [ + 1046, + 3754, + 13025, + 275, + 3345, + 14510, + 396, + 75040, + 103, + 872, + 304, + 423, + 108364, + 27, + 373, + 1633, + 312, + 1730, + 275, + 1259, + 76, + 64301, + 2122, + 478, + 275, + 6594, + 518, + 64, + 872, + 103, + 872, + 15, + 317, + 9261, + 518, + 8281, + 4348, + 198, + 198, + 13104, + 713, + 2426, + 275, + 2122, + 27, + 4, + 54, + 7632, + 1259, + 275, + 3345, + 14510, + 396, + 75040, + 103, + 872, + 2430, + 2343, + 27, + 130977, + 913, + 76, + 64301, + 130980 + ] + }, + "plain_0": { + "text": " the capital of Australia.\nassistant\nThe user is asking me to act as an assistant. I need to understand the context. The user's current message is: \"The user is asking me to act as an assistant. I need to understand the context. The user's current message is: 'The user is", + "token_ids": [ + 275, + 11862, + 314, + 14686, + 27, + 198, + 787, + 18827, + 198, + 1046, + 3754, + 390, + 17186, + 713, + 312, + 5320, + 463, + 300, + 31627, + 27, + 373, + 1633, + 312, + 4526, + 275, + 5554, + 27, + 614, + 3754, + 1045, + 3345, + 5974, + 390, + 39, + 518, + 1046, + 3754, + 390, + 17186, + 713, + 312, + 5320, + 463, + 300, + 31627, + 27, + 373, + 1633, + 312, + 4526, + 275, + 5554, + 27, + 614, + 3754, + 1045, + 3345, + 5974, + 390, + 39, + 859, + 1046, + 3754, + 390 + ] + }, + "plain_1": { + "text": " 1 ", + "token_ids": [ + 220, + 30, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220 + ] + }, + "plain_2": { + "text": " return [1, 1] + [0] + [1] * (n - 2)\n\n\n# Test the Fibonacci function.\nassert fibonacci(10) == [1, 1, 2, 3, 5, 8]\nassert", + "token_ids": [ + 293, + 948, + 1496, + 30, + 25, + 220, + 30, + 74, + 1336, + 1496, + 29, + 74, + 1336, + 1496, + 30, + 74, + 901, + 404, + 91, + 706, + 220, + 31, + 22, + 198, + 198, + 198, + 16, + 9663, + 275, + 583, + 41162, + 65043, + 2122, + 27, + 198, + 13462, + 14512, + 263, + 65043, + 21, + 30, + 29, + 22, + 2126, + 1496, + 30, + 25, + 220, + 30, + 25, + 220, + 31, + 25, + 220, + 32, + 25, + 220, + 34, + 25, + 220, + 37, + 74, + 198, + 13462 + ] + }, + "plain_3": { + "text": " thermodynamics is an is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is", + "token_ids": [ + 13294, + 121223, + 390, + 300, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390 + ] + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": [ + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312 + ] + }, + "plain_5": { + "text": " is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is", + "token_ids": [ + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390 + ] + }, + "plain_6": { + "text": " into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into", + "token_ids": [ + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464 + ] + }, + "plain_7": { + "text": " is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that", + "token_ids": [ + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455 + ] + }, + "plain_8": { + "text": " Westphalia. Westphalia was a region of Germany.\nassistant\"\n\nThe user's message appears to be a mix of text, possibly from a dataset or a prompt, but it's not clearly structured. I need to understand what the user wants me to do. Since the user's message is", + "token_ids": [ + 10519, + 1672, + 14288, + 27, + 10519, + 1672, + 14288, + 572, + 259, + 6987, + 314, + 19356, + 27, + 198, + 787, + 18827, + 15, + 198, + 198, + 1046, + 3754, + 1045, + 5974, + 12718, + 312, + 417, + 259, + 12213, + 314, + 4217, + 25, + 16809, + 692, + 259, + 23977, + 553, + 259, + 20207, + 25, + 961, + 488, + 1045, + 702, + 12269, + 42689, + 27, + 373, + 1633, + 312, + 4526, + 1568, + 275, + 3754, + 13025, + 713, + 312, + 793, + 27, + 11342, + 275, + 3754, + 1045, + 5974, + 390 + ] + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/README.md b/benchmarks/edge_harness/results/README.md new file mode 100644 index 000000000..7e29fb30f --- /dev/null +++ b/benchmarks/edge_harness/results/README.md @@ -0,0 +1,32 @@ +# Result artifacts + +Raw output backing the published figures. Every file written by the current +harness carries a `provenance` block — model path, quantization, thread +affinity, build SHAs, host, environment — so a number can be traced to the run +that produced it and invalidated later if the run turns out to be wrong. + +```bash +python -c "import json,sys; print(json.load(open(sys.argv[1]))['provenance'])" .json +``` + +## Naming + +| prefix | contents | +|---|---| +| `profile_*` | per-op decode profiles (`profile_cpu_step.py`) | +| `parity_*`, `export_parity_*` | numerical parity checks against a reference | +| `quality_*`, `cmp_*` | greedy-divergence fidelity | +| `aihub_*` | Qualcomm AI Hub on-device runs (Snapdragon) | +| `*_grid.jsonl` | interleaved A/B grids | +| `s25_*` | Galaxy S25 device estimates | + +## Retracted files are kept, not deleted + +Anything with `"retracted": true` names the reason and its replacement. +`profile_w4a8_g128_p{1,2}.json` are the clearest case: both show +`_C::onednn_mm` at ~2.97 ms/step — the signature of a checkpoint whose output +head was left in bf16 — and every W4A8 number derived from them is void. The +valid replacement is `profile_w4a8_g128_head_p1.json`. + +They are kept because a deleted mistake teaches nothing, and because the older +reports still cite them. diff --git a/benchmarks/edge_harness/results/ab_grid.jsonl b/benchmarks/edge_harness/results/ab_grid.jsonl new file mode 100644 index 000000000..3666aa9a2 --- /dev/null +++ b/benchmarks/edge_harness/results/ab_grid.jsonl @@ -0,0 +1,6 @@ +{"tag": "base", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 83.15, "rss_mb": 8663.0, "rss_peak_mb": 8699.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2227, "pp_tps": 2299.08, "tg_tokens": 128, "tg_s": 2.2158, "tg_tps": 57.32, "raw": {"pp": [0.31149862613528967, 0.3030338645912707, 0.22269816184416413], "tg": [2.3680049385875463, 2.3106129062362015, 2.229406553786248], "tg1": [0.01362658105790615, 0.013699103146791458, 0.013955966103821993]}} +{"tag": "gate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 50.46, "rss_mb": 8681.6, "rss_peak_mb": 8710.0, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1508, "pp_tps": 3394.1, "tg_tokens": 128, "tg_s": 1.6888, "tg_tps": 75.2, "raw": {"pp": [0.16184503305703402, 0.1508498010225594, 0.15504808397963643], "tg": [1.7033594409003854, 1.8424865039996803, 2.1489964802749455], "tg1": [0.016280516982078552, 0.015342713799327612, 0.014542239718139172]}} +{"tag": "gate+chunk", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": "{\"inductor_compile_config\":{\"cpp.min_chunk_size\":65536}}", "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 80.04, "rss_mb": 8612.2, "rss_peak_mb": 8640.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1866, "pp_tps": 2743.17, "tg_tokens": 128, "tg_s": 1.8041, "tg_tps": 70.4, "raw": {"pp": [0.2269656560383737, 0.19936348916962743, 0.18664512783288956], "tg": [1.8515523397363722, 1.8183003733865917, 1.8471249351277947], "tg1": [0.014217090792953968, 0.016116315964609385, 0.01467960188165307]}} +{"tag": "gate+chunk+g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": "{\"inductor_compile_config\":{\"cpp.min_chunk_size\":65536}}", "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 86.06, "rss_mb": 8056.8, "rss_peak_mb": 8084.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1512, "pp_tps": 3387.19, "tg_tokens": 128, "tg_s": 1.6258, "tg_tps": 78.12, "raw": {"pp": [0.1577955186367035, 0.15690755285322666, 0.15115766506642103], "tg": [1.690211537759751, 1.6389126512221992, 1.6603844622150064], "tg1": [0.013693444896489382, 0.013548826798796654, 0.013116502668708563]}} +{"tag": "w4a8+gate+chunk", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": "{\"inductor_compile_config\":{\"cpp.min_chunk_size\":65536}}", "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 79.15, "rss_mb": 9519.8, "rss_peak_mb": 9541.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2452, "pp_tps": 2088.2, "tg_tokens": 128, "tg_s": 2.2477, "tg_tps": 56.5, "raw": {"pp": [0.24662079522386193, 0.2584368041716516, 0.24518772400915623], "tg": [2.265108836814761, 2.3531849440187216, 2.368000926449895], "tg1": [0.01957006100565195, 0.019077030941843987, 0.017433146946132183]}} +{"tag": "base-repeat", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.8, "rss_mb": 8724.6, "rss_peak_mb": 8766.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2006, "pp_tps": 2551.88, "tg_tokens": 128, "tg_s": 1.8666, "tg_tps": 68.04, "raw": {"pp": [0.23906681034713984, 0.2106484593823552, 0.20063663413748145], "tg": [1.9647751650772989, 1.9212167048826814, 1.8819844778627157], "tg1": [0.016026358120143414, 0.0173974996432662, 0.015383167192339897]}} diff --git a/benchmarks/edge_harness/results/aihub_block4.json b/benchmarks/edge_harness/results/aihub_block4.json new file mode 100644 index 000000000..1a64df87a --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_block4.json @@ -0,0 +1,3 @@ +{ + "fp16": 6.103 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_calib_full.json b/benchmarks/edge_harness/results/aihub_calib_full.json new file mode 100644 index 000000000..ad985d4d3 --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_calib_full.json @@ -0,0 +1,10 @@ +{ + "w8a16": { + "snr_db": 37.39, + "ms": 0.942 + }, + "w16a16": { + "snr_db": 44.73, + "ms": 1.656 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_calib_lmhead.json b/benchmarks/edge_harness/results/aihub_calib_lmhead.json new file mode 100644 index 000000000..93d14b876 --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_calib_lmhead.json @@ -0,0 +1,10 @@ +{ + "w8a16": { + "snr_db": 62.76, + "ms": 3.907 + }, + "w16a16": { + "snr_db": 87.44, + "ms": 7.471 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_calibrated.json b/benchmarks/edge_harness/results/aihub_calibrated.json new file mode 100644 index 000000000..541a5bdb7 --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_calibrated.json @@ -0,0 +1,10 @@ +{ + "w8a16": { + "snr_db": 36.85, + "ms": 0.803 + }, + "w8a8": { + "snr_db": 5.06, + "ms": 0.749 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_cpu_int8.json b/benchmarks/edge_harness/results/aihub_cpu_int8.json new file mode 100644 index 000000000..55945483b --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_cpu_int8.json @@ -0,0 +1,3 @@ +{ + "tflite_int8_cpu": 0.877 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_gelu.json b/benchmarks/edge_harness/results/aihub_gelu.json new file mode 100644 index 000000000..ff83c524b --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_gelu.json @@ -0,0 +1,3 @@ +{ + "ring_sliding_tanh_fp16": 1.535 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_int4.json b/benchmarks/edge_harness/results/aihub_int4.json new file mode 100644 index 000000000..198330ce0 --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_int4.json @@ -0,0 +1,10 @@ +{ + "ctx_no_qio": { + "ms": null, + "snr_db": null + }, + "ctx_qio": { + "ms": null, + "snr_db": null + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_parity_a8.json b/benchmarks/edge_harness/results/aihub_parity_a8.json new file mode 100644 index 000000000..1354aa82b --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_parity_a8.json @@ -0,0 +1,5 @@ +{ + "fp16": 60.63, + "w4a16": -1.6, + "w4a8": -1.28 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_processors.json b/benchmarks/edge_harness/results/aihub_processors.json new file mode 100644 index 000000000..b6e71ece8 --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_processors.json @@ -0,0 +1,12 @@ +{ + "ring_sliding_ctx1024|qnn_npu": 1.479, + "ring_sliding_ctx1024|tflite_npu": 1.491, + "ring_sliding_ctx1024|tflite_cpu": 2.958, + "ring_sliding_ctx1024|tflite_gpu": 2.259, + "ring_sliding_ctx1024|onnx_npu": 1.5, + "ring_full_ctx1024|qnn_npu": 1.554, + "ring_full_ctx1024|tflite_npu": 1.578, + "ring_full_ctx1024|tflite_cpu": 3.006, + "ring_full_ctx1024|tflite_gpu": 2.632, + "ring_full_ctx1024|onnx_npu": 1.594 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_ring.json b/benchmarks/edge_harness/results/aihub_ring.json new file mode 100644 index 000000000..01708a04e --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_ring.json @@ -0,0 +1,6 @@ +{ + "ring_sliding_ctx1024|fp16": 1.483, + "ring_full_ctx1024|fp16": 1.567, + "lm_head|w4a16": 3.868, + "lm_head|w8a16": 3.862 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_ring2.json b/benchmarks/edge_harness/results/aihub_ring2.json new file mode 100644 index 000000000..8cba5f45f --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_ring2.json @@ -0,0 +1,3 @@ +{ + "fp16": 1.48 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_ring3.json b/benchmarks/edge_harness/results/aihub_ring3.json new file mode 100644 index 000000000..6ed2499e0 --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_ring3.json @@ -0,0 +1,4 @@ +{ + "w4a16": 0.838, + "fp16": 1.486 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_spark_layers.json b/benchmarks/edge_harness/results/aihub_spark_layers.json new file mode 100644 index 000000000..18c82ea01 --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_spark_layers.json @@ -0,0 +1,11 @@ +{ + "sliding|ctx1024|Samsung Galaxy S25|fp16": 1.689, + "sliding|ctx1024|Samsung Galaxy S25|w8a16": 0.912, + "sliding|ctx1024|Samsung Galaxy S25|w4a16": 0.917, + "full|ctx1024|Samsung Galaxy S25|fp16": 1.559, + "full|ctx1024|Samsung Galaxy S25|w8a16": 1.129, + "full|ctx1024|Samsung Galaxy S25|w4a16": 1.127, + "full|ctx4096|Samsung Galaxy S25|fp16": 2.964, + "full|ctx4096|Samsung Galaxy S25|w8a16": 1.681, + "full|ctx4096|Samsung Galaxy S25|w4a16": 1.662 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_spark_runtimes.json b/benchmarks/edge_harness/results/aihub_spark_runtimes.json new file mode 100644 index 000000000..3db84150d --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_spark_runtimes.json @@ -0,0 +1,7 @@ +{ + "qnn_npu": 1.563, + "tflite_npu": 1.579, + "onnx_npu": 1.594, + "tflite_gpu": 3.752, + "tflite_cpu": 3.014 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/aihub_w8a8.json b/benchmarks/edge_harness/results/aihub_w8a8.json new file mode 100644 index 000000000..6a26c8669 --- /dev/null +++ b/benchmarks/edge_harness/results/aihub_w8a8.json @@ -0,0 +1,3 @@ +{ + "w4a8": 0.751 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/batch_bench.jsonl b/benchmarks/edge_harness/results/batch_bench.jsonl new file mode 100644 index 000000000..e78c42838 --- /dev/null +++ b/benchmarks/edge_harness/results/batch_bench.jsonl @@ -0,0 +1,3 @@ +{"tag": "vllm-bf16-t32", "threads": "32", "gen": 128, "prompt": 128, "tok_s_by_batch": {"1": 34.56, "2": 69.87, "4": 139.76, "8": 266.13, "16": 495.67}} +{"tag": "vllm-int8-t32", "threads": "32", "gen": 128, "prompt": 128, "tok_s_by_batch": {"1": 54.87, "2": 105.21, "4": 210.3, "8": 387.84, "16": 694.7}} +{"tag": "vllm-int8-t32-hi", "threads": "32", "gen": 128, "prompt": 128, "tok_s_by_batch": {"32": 1080.59, "64": 1557.9}} diff --git a/benchmarks/edge_harness/results/export_parity.json b/benchmarks/edge_harness/results/export_parity.json new file mode 100644 index 000000000..f0b939cb6 --- /dev/null +++ b/benchmarks/edge_harness/results/export_parity.json @@ -0,0 +1,14 @@ +{ + "context": 1024, + "logits_snr_db": 138.54, + "max_abs_diff": 1.9073486328125e-05, + "ref_top1": 1310, + "export_top1": 1310, + "top1_agree": true, + "top5_agree": true, + "kv_bytes": { + "hybrid": 36700160, + "uniform": 58720256, + "saved": 22020096 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/export_parity_auto.json b/benchmarks/edge_harness/results/export_parity_auto.json new file mode 100644 index 000000000..1c8691cca --- /dev/null +++ b/benchmarks/edge_harness/results/export_parity_auto.json @@ -0,0 +1,16 @@ +{ + "context": 1024, + "cache_layout": "auto", + "gelu_mode": "exact", + "logits_snr_db": 134.84, + "max_abs_diff": 2.288818359375e-05, + "ref_top1": 1310, + "export_top1": 1310, + "top1_agree": true, + "top5_agree": true, + "kv_bytes": { + "hybrid": 36657152, + "uniform": 58720256, + "saved": 22063104 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/export_parity_gelu_exact.json b/benchmarks/edge_harness/results/export_parity_gelu_exact.json new file mode 100644 index 000000000..36c232510 --- /dev/null +++ b/benchmarks/edge_harness/results/export_parity_gelu_exact.json @@ -0,0 +1,16 @@ +{ + "context": 1024, + "cache_layout": "ring", + "gelu_mode": "exact", + "logits_snr_db": 134.93, + "max_abs_diff": 1.9073486328125e-05, + "ref_top1": 1310, + "export_top1": 1310, + "top1_agree": true, + "top5_agree": true, + "kv_bytes": { + "hybrid": 36657152, + "uniform": 58720256, + "saved": 22063104 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/export_parity_gelu_sigmoid.json b/benchmarks/edge_harness/results/export_parity_gelu_sigmoid.json new file mode 100644 index 000000000..a6bf363a8 --- /dev/null +++ b/benchmarks/edge_harness/results/export_parity_gelu_sigmoid.json @@ -0,0 +1,16 @@ +{ + "context": 1024, + "cache_layout": "ring", + "gelu_mode": "sigmoid", + "logits_snr_db": 52.35, + "max_abs_diff": 0.3539543151855469, + "ref_top1": 1310, + "export_top1": 1310, + "top1_agree": true, + "top5_agree": true, + "kv_bytes": { + "hybrid": 36657152, + "uniform": 58720256, + "saved": 22063104 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/export_parity_gelu_tanh.json b/benchmarks/edge_harness/results/export_parity_gelu_tanh.json new file mode 100644 index 000000000..a8513de06 --- /dev/null +++ b/benchmarks/edge_harness/results/export_parity_gelu_tanh.json @@ -0,0 +1,16 @@ +{ + "context": 1024, + "cache_layout": "ring", + "gelu_mode": "tanh", + "logits_snr_db": 65.24, + "max_abs_diff": 0.028232574462890625, + "ref_top1": 1310, + "export_top1": 1310, + "top1_agree": true, + "top5_agree": true, + "kv_bytes": { + "hybrid": 36657152, + "uniform": 58720256, + "saved": 22063104 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/export_parity_ring.json b/benchmarks/edge_harness/results/export_parity_ring.json new file mode 100644 index 000000000..68cf791a6 --- /dev/null +++ b/benchmarks/edge_harness/results/export_parity_ring.json @@ -0,0 +1,16 @@ +{ + "context": 1024, + "cache_layout": "ring", + "gelu_mode": "exact", + "logits_snr_db": 139.05, + "max_abs_diff": 1.9073486328125e-05, + "ref_top1": 1310, + "export_top1": 1310, + "top1_agree": true, + "top5_agree": true, + "kv_bytes": { + "hybrid": 36657152, + "uniform": 58720256, + "saved": 22063104 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/export_parity_ring2.json b/benchmarks/edge_harness/results/export_parity_ring2.json new file mode 100644 index 000000000..ec3b257d1 --- /dev/null +++ b/benchmarks/edge_harness/results/export_parity_ring2.json @@ -0,0 +1,16 @@ +{ + "context": 1024, + "cache_layout": "ring", + "gelu_mode": "exact", + "logits_snr_db": 136.84, + "max_abs_diff": 2.288818359375e-05, + "ref_top1": 1310, + "export_top1": 1310, + "top1_agree": true, + "top5_agree": true, + "kv_bytes": { + "hybrid": 36657152, + "uniform": 58720256, + "saved": 22063104 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/fusion_ab.jsonl b/benchmarks/edge_harness/results/fusion_ab.jsonl new file mode 100644 index 000000000..e00019c00 --- /dev/null +++ b/benchmarks/edge_harness/results/fusion_ab.jsonl @@ -0,0 +1,8 @@ +{"tag": "w4a8-g128-nofuse", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 52.13, "rss_mb": 8453.0, "rss_peak_mb": 8476.7, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1522, "pp_tps": 3364.23, "tg_tokens": 128, "tg_s": 1.7271, "tg_tps": 73.53, "raw": {"pp": [0.17440469283610582, 0.17541490076109767, 0.1521894964389503], "tg": [1.7558523141779006, 1.741288021672517, 1.7787284357473254], "tg1": [0.016926162876188755, 0.01439615711569786, 0.014146057888865471]}} +{"tag": "w4a16-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 68.51, "rss_mb": 8042.3, "rss_peak_mb": 8070.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1505, "pp_tps": 3402.93, "tg_tokens": 128, "tg_s": 1.6184, "tg_tps": 78.47, "raw": {"pp": [0.15815730998292565, 0.15045850770547986, 0.15534949069842696], "tg": [1.649985218886286, 1.6359889470040798, 1.6321744010783732], "tg1": [0.013913698960095644, 0.013769291341304779, 0.01399048138409853]}} +{"tag": "w4a8-g128-nofuse", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 14.94, "rss_mb": 8319.6, "rss_peak_mb": 8352.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1528, "pp_tps": 3350.83, "tg_tokens": 128, "tg_s": 1.7166, "tg_tps": 73.98, "raw": {"pp": [0.1551546179689467, 0.15429807314649224, 0.15279806219041348], "tg": [1.8323531011119485, 1.7875734940171242, 1.7306284229271114], "tg1": [0.014606636948883533, 0.015294387936592102, 0.013988425955176353]}} +{"tag": "w4a8-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 64.55, "rss_mb": 8418.2, "rss_peak_mb": 8441.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1477, "pp_tps": 3465.57, "tg_tokens": 128, "tg_s": 1.7468, "tg_tps": 72.71, "raw": {"pp": [0.1485598487779498, 0.14842529594898224, 0.14773900294676423], "tg": [1.771411261986941, 1.828272869810462, 1.7606302723288536], "tg1": [0.013911445159465075, 0.013920526020228863, 0.01385196065530181]}} +{"tag": "w4a16-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 12.96, "rss_mb": 7919.4, "rss_peak_mb": 7946.8, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.15, "pp_tps": 3412.23, "tg_tokens": 128, "tg_s": 1.5981, "tg_tps": 79.47, "raw": {"pp": [0.15004837000742555, 0.15867038909345865, 0.15092349611222744], "tg": [1.640803290065378, 1.6112936749123037, 1.6191998501308262], "tg1": [0.013224144000560045, 0.013160659931600094, 0.0131501080468297]}} +{"tag": "w4a8-g128-nofuse", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 14.72, "rss_mb": 8316.2, "rss_peak_mb": 8346.6, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1501, "pp_tps": 3410.0, "tg_tokens": 128, "tg_s": 1.7373, "tg_tps": 73.1, "raw": {"pp": [0.1501467190682888, 0.15637517208233476, 0.1522511402145028], "tg": [1.7511884178966284, 1.76484130974859, 1.7585765281692147], "tg1": [0.013923108112066984, 0.01469977805390954, 0.014159561600536108]}} +{"tag": "w4a8-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 14.96, "rss_mb": 8282.7, "rss_peak_mb": 8342.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.152, "pp_tps": 3368.08, "tg_tokens": 128, "tg_s": 1.846, "tg_tps": 68.8, "raw": {"pp": [0.15289765782654285, 0.15201554354280233, 0.1526965661905706], "tg": [1.8671486796811223, 1.8718066262081265, 1.8615437718108296], "tg1": [0.01563759008422494, 0.018718563951551914, 0.015583970118314028]}} +{"tag": "w4a16-g128-fused", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 12.84, "rss_mb": 7918.4, "rss_peak_mb": 7946.1, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1591, "pp_tps": 3218.41, "tg_tokens": 128, "tg_s": 1.59, "tg_tps": 79.87, "raw": {"pp": [0.1617804798297584, 0.15908468374982476, 0.22660941490903497], "tg": [1.6176169221289456, 1.6027018236927688, 1.6187863657251], "tg1": [0.012814681977033615, 0.012712633702903986, 0.01277841068804264]}} diff --git a/benchmarks/edge_harness/results/gpu_bench.jsonl b/benchmarks/edge_harness/results/gpu_bench.jsonl new file mode 100644 index 000000000..d27c7998e --- /dev/null +++ b/benchmarks/edge_harness/results/gpu_bench.jsonl @@ -0,0 +1,2 @@ +{"engine": "vllm", "dtype": "bfloat16", "gpu": "L20X", "init_s": 120.21, "pp_tokens": 512, "pp_s": 0.0116, "pp_tps": 44054.08, "tg_tokens": 128, "tg_s": 0.2767, "tg_tps": 458.97} +{"engine": "sglang", "dtype": "bfloat16", "gpu": "L20X", "init_s": 102.77, "pp_tokens": 512, "pp_s": 0.0101, "pp_tps": 50934.7, "tg_tokens": 128, "tg_s": 0.2902, "tg_tps": 437.68} diff --git a/benchmarks/edge_harness/results/gpu_bench_pp2048.jsonl b/benchmarks/edge_harness/results/gpu_bench_pp2048.jsonl new file mode 100644 index 000000000..8647756f8 --- /dev/null +++ b/benchmarks/edge_harness/results/gpu_bench_pp2048.jsonl @@ -0,0 +1,2 @@ +{"engine": "vllm", "dtype": "bfloat16", "gpu": "L20X", "init_s": 111.25, "pp_tokens": 2048, "pp_s": 0.0149, "pp_tps": 137102.32, "tg_tokens": 128, "tg_s": 0.2813, "tg_tps": 451.4} +{"engine": "sglang", "dtype": "bfloat16", "gpu": "L20X", "init_s": 97.05, "pp_tokens": 2048, "pp_s": 0.0144, "pp_tps": 142252.89, "tg_tokens": 128, "tg_s": 0.2921, "tg_tps": 434.72} diff --git a/benchmarks/edge_harness/results/init_memory.jsonl b/benchmarks/edge_harness/results/init_memory.jsonl new file mode 100644 index 000000000..5274c1697 --- /dev/null +++ b/benchmarks/edge_harness/results/init_memory.jsonl @@ -0,0 +1,6 @@ +{"tag": "cold-shim-on", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 93.77, "rss_mb": 8502.3, "rss_peak_mb": 8502.6, "maxrss_mb": 8502.6, "kvcache_space_gib": "4", "cache_root": "/tmp/claude-2005/-data-zhoutaichang-embedding-infer/e695f57b-0163-4287-b1be-1d0ec9330768/scratchpad/vllm_cache_fresh_1870348", "aot_shim": "1", "max_model_len": 2048, "tokens_generated": 8} +{"tag": "warm-shim-on", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 14.87, "rss_mb": 8335.3, "rss_peak_mb": 8346.9, "maxrss_mb": 8346.9, "kvcache_space_gib": "4", "cache_root": "/tmp/claude-2005/-data-zhoutaichang-embedding-infer/e695f57b-0163-4287-b1be-1d0ec9330768/scratchpad/vllm_cache_fresh_1870348", "aot_shim": "1", "max_model_len": 2048, "tokens_generated": 8} +{"tag": "warm-shim-off", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 49.83, "rss_mb": 8473.4, "rss_peak_mb": 8473.4, "maxrss_mb": 8473.4, "kvcache_space_gib": "4", "cache_root": "/tmp/claude-2005/-data-zhoutaichang-embedding-infer/e695f57b-0163-4287-b1be-1d0ec9330768/scratchpad/vllm_cache_fresh_1870348", "aot_shim": "0", "max_model_len": 2048, "tokens_generated": 8} +{"tag": "mem-kv4gib", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 15.11, "rss_mb": 8329.1, "rss_peak_mb": 8356.6, "maxrss_mb": 8356.6, "kvcache_space_gib": "4", "kv_cache_memory_bytes": null, "cache_root": "(default)", "aot_shim": "1", "max_model_len": 2048, "tokens_generated": 8} +{"tag": "mem-kv128mib", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 14.24, "rss_mb": 5254.5, "rss_peak_mb": 5359.9, "maxrss_mb": 5359.9, "kvcache_space_gib": "1", "kv_cache_memory_bytes": 134217728, "cache_root": "(default)", "aot_shim": "1", "max_model_len": 2048, "tokens_generated": 8} +{"tag": "mem-kv64mib", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "init_s": 76.57, "rss_mb": 5394.2, "rss_peak_mb": 5394.4, "maxrss_mb": 5394.4, "kvcache_space_gib": "1", "kv_cache_memory_bytes": 67108864, "cache_root": "(default)", "aot_shim": "1", "max_model_len": 1024, "tokens_generated": 8} diff --git a/benchmarks/edge_harness/results/int4_grid.jsonl b/benchmarks/edge_harness/results/int4_grid.jsonl new file mode 100644 index 000000000..a17376da8 --- /dev/null +++ b/benchmarks/edge_harness/results/int4_grid.jsonl @@ -0,0 +1,5 @@ +{"tag": "vllm-int4-g64-t4", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 81.24, "threads_bind": "0-3", "omp_num_threads": "4", "pp_tokens": 512, "pp_s": 0.5824, "pp_tps": 879.12, "tg_tokens": 128, "tg_s": 4.7788, "tg_tps": 26.58, "raw": {"pp": [0.588689471129328, 0.5823977291584015, 0.5908681848086417], "tg": [4.8179576066322625, 5.019054782111198, 6.836774786002934], "tg1": [0.039164267014712095, 0.04048056015744805, 0.03950501000508666]}} +{"tag": "vllm-int4-g64-t8", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 79.16, "threads_bind": "0-7", "omp_num_threads": "8", "pp_tokens": 512, "pp_s": 0.321, "pp_tps": 1595.07, "tg_tokens": 128, "tg_s": 2.8446, "tg_tps": 44.65, "raw": {"pp": [0.3280641417950392, 0.32098965905606747, 0.33939772471785545], "tg": [2.875607998110354, 2.867094303946942, 2.866884418297559], "tg1": [0.02253475086763501, 0.022273391019552946, 0.022320172749459743]}} +{"tag": "vllm-int4-g64-t16", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 46.81, "threads_bind": "0-15", "omp_num_threads": "16", "pp_tokens": 512, "pp_s": 0.189, "pp_tps": 2708.83, "tg_tokens": 128, "tg_s": 1.9166, "tg_tps": 66.26, "raw": {"pp": [0.1913696713745594, 0.2119541959837079, 0.18901151278987527], "tg": [1.9710817309096456, 1.9320780518464744, 1.9449053118005395], "tg1": [0.015898379031568766, 0.015468112193048, 0.015775869134813547]}} +{"tag": "vllm-int4-g64-t32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 47.57, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1502, "pp_tps": 3408.73, "tg_tokens": 128, "tg_s": 1.6334, "tg_tps": 77.75, "raw": {"pp": [0.18242419138550758, 0.15723237209022045, 0.1502025001682341], "tg": [1.6569905988872051, 1.646598860155791, 1.6571382582187653], "tg1": [0.013176599983125925, 0.013634935952723026, 0.013306272216141224]}} +{"tag": "vllm-bf16-t32-control", "model": "XHToken/Spark-X2.5-1.7B", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 78.93, "rss_mb": 10394.9, "rss_peak_mb": 10430.2, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1068, "pp_tps": 4795.53, "tg_tokens": 128, "tg_s": 3.0986, "tg_tps": 40.99, "raw": {"pp": [0.11023258324712515, 0.10676615126430988, 0.1215914050117135], "tg": [3.1550607047975063, 3.1232189969159663, 3.155732364859432], "tg1": [0.02565043093636632, 0.02529451809823513, 0.024592541623860598]}} diff --git a/benchmarks/edge_harness/results/interleaved.jsonl b/benchmarks/edge_harness/results/interleaved.jsonl new file mode 100644 index 000000000..7c2b4682a --- /dev/null +++ b/benchmarks/edge_harness/results/interleaved.jsonl @@ -0,0 +1,14 @@ +{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 57.41, "rss_mb": 8670.4, "rss_peak_mb": 8702.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2557, "pp_tps": 2002.6, "tg_tokens": 128, "tg_s": 2.3198, "tg_tps": 54.75, "raw": {"pp": [0.25566731160506606, 0.2593384999781847, 0.2671640901826322], "tg": [2.368054673075676, 2.3595390100963414, 2.337980422191322], "tg1": [0.021325419656932354, 0.018553890753537416, 0.01813417114317417]}} +{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 52.38, "rss_mb": 8709.2, "rss_peak_mb": 8729.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.222, "pp_tps": 2306.35, "tg_tokens": 128, "tg_s": 2.0763, "tg_tps": 61.17, "raw": {"pp": [0.2219955581240356, 0.25641375267878175, 0.25716910790652037], "tg": [2.398225273936987, 2.0967630273662508, 2.0924383951351047], "tg1": [0.01652884017676115, 0.016112301964312792, 0.017952477093786]}} +{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 87.37, "rss_mb": 8062.0, "rss_peak_mb": 8089.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1671, "pp_tps": 3064.11, "tg_tokens": 128, "tg_s": 1.9204, "tg_tps": 66.13, "raw": {"pp": [0.16709583811461926, 0.1969963563606143, 0.2156481253914535], "tg": [2.0638905209489167, 1.9353438159450889, 2.022206211928278], "tg1": [0.014969798736274242, 0.015494609717279673, 0.01520752115175128]}} +{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 84.05, "rss_mb": 9511.1, "rss_peak_mb": 9533.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2651, "pp_tps": 1931.13, "tg_tokens": 128, "tg_s": 1.9723, "tg_tps": 64.39, "raw": {"pp": [0.26512908516451716, 0.2868352457880974, 0.2711168541572988], "tg": [2.0084553682245314, 1.9880874049849808, 2.059911102987826], "tg1": [0.019026058726012707, 0.015752890147268772, 0.015950628090649843]}} +{"tag": "w4a8-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 82.16, "rss_mb": 8097.7, "rss_peak_mb": 9061.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.153, "pp_tps": 3346.62, "tg_tokens": 128, "tg_s": 1.5022, "tg_tps": 84.54, "raw": {"pp": [0.1572714359499514, 0.1576147861778736, 0.15299037797376513], "tg": [1.5144385821186006, 1.5316730751655996, 1.522148437332362], "tg1": [0.012234935071319342, 0.01230821618810296, 0.012677683960646391]}} +{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.18, "rss_mb": 8673.2, "rss_peak_mb": 8712.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2556, "pp_tps": 2002.96, "tg_tokens": 128, "tg_s": 1.8377, "tg_tps": 69.11, "raw": {"pp": [0.27302785124629736, 0.2624788419343531, 0.255622121039778], "tg": [2.8208564538508654, 1.8630885141901672, 2.3649882823228836], "tg1": [0.025365081150084734, 0.026843297760933638, 0.029472910799086094]}} +{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.75, "rss_mb": 8686.3, "rss_peak_mb": 8714.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1596, "pp_tps": 3207.28, "tg_tokens": 128, "tg_s": 1.6919, "tg_tps": 75.06, "raw": {"pp": [0.16286137886345387, 0.16863904893398285, 0.15963682811707258], "tg": [1.71276545105502, 1.704967309255153, 1.748305486049503], "tg1": [0.0130898617208004, 0.013152792118489742, 0.013696761801838875]}} +{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 48.85, "rss_mb": 8064.4, "rss_peak_mb": 8092.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1581, "pp_tps": 3238.56, "tg_tokens": 128, "tg_s": 1.6414, "tg_tps": 77.37, "raw": {"pp": [0.20444863196462393, 0.16155556635931134, 0.1580951907671988], "tg": [1.654232523869723, 1.6747960466891527, 1.668766028713435], "tg1": [0.012994957156479359, 0.012909624725580215, 0.012812163680791855]}} +{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.65, "rss_mb": 9512.0, "rss_peak_mb": 9535.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.224, "pp_tps": 2285.48, "tg_tokens": 128, "tg_s": 1.584, "tg_tps": 80.18, "raw": {"pp": [0.22863413486629725, 0.22572151478379965, 0.2240230217576027], "tg": [1.5968111460097134, 1.6221620081923902, 1.6093115280382335], "tg1": [0.013640933204442263, 0.012812962755560875, 0.01369112217798829]}} +{"tag": "w4a16-g64-nogate", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.55, "rss_mb": 8657.9, "rss_peak_mb": 8694.2, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.7995, "pp_tps": 640.36, "tg_tokens": 128, "tg_s": 13.9584, "tg_tps": 9.1, "raw": {"pp": [0.8044024519622326, 0.8655711049214005, 0.7995461183600128], "tg": [14.970092439092696, 14.903033270966262, 13.9971464141272], "tg1": [0.18624361278489232, 0.03878649603575468, 0.08108040131628513]}} +{"tag": "w4a16-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.91, "rss_mb": 8696.9, "rss_peak_mb": 8724.5, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1576, "pp_tps": 3248.3, "tg_tokens": 128, "tg_s": 1.7837, "tg_tps": 71.2, "raw": {"pp": [0.16414545476436615, 0.15762093104422092, 0.16772186197340488], "tg": [1.7981987767852843, 1.8183860406279564, 1.8294021431356668], "tg1": [0.014506598934531212, 0.01543053099885583, 0.01460421597585082]}} +{"tag": "w4a16-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 49.34, "rss_mb": 8057.9, "rss_peak_mb": 8084.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.153, "pp_tps": 3345.68, "tg_tokens": 128, "tg_s": 1.6146, "tg_tps": 78.66, "raw": {"pp": [0.15356025705114007, 0.15303313126787543, 0.1586752710863948], "tg": [1.6407190039753914, 1.6276185680180788, 1.6660430012270808], "tg1": [0.012987045105546713, 0.01297468226402998, 0.013056765776127577]}} +{"tag": "w4a8-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 53.23, "rss_mb": 9514.0, "rss_peak_mb": 9536.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.2323, "pp_tps": 2203.64, "tg_tokens": 128, "tg_s": 1.788, "tg_tps": 71.03, "raw": {"pp": [0.23234243784099817, 0.258395710028708, 0.2591813113540411], "tg": [1.8015480861067772, 1.803529669996351, 1.8111027288250625], "tg1": [0.014497074764221907, 0.01430785097181797, 0.013512391597032547]}} +{"tag": "w4a8-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 79.0, "rss_mb": 8461.2, "rss_peak_mb": 8485.7, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1784, "pp_tps": 2869.4, "tg_tokens": 128, "tg_s": 1.953, "tg_tps": 65.03, "raw": {"pp": [0.1784344231709838, 0.19405587995424867, 0.1978841326199472], "tg": [2.0177465733140707, 1.9670254928059876, 2.016505080740899], "tg1": [0.015316586010158062, 0.014635276980698109, 0.014058694709092379]}} diff --git a/benchmarks/edge_harness/results/leader_knobs.jsonl b/benchmarks/edge_harness/results/leader_knobs.jsonl new file mode 100644 index 000000000..43d0ddd22 --- /dev/null +++ b/benchmarks/edge_harness/results/leader_knobs.jsonl @@ -0,0 +1,9 @@ +{"tag": "leader", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 53.99, "rss_mb": 8470.9, "rss_peak_mb": 8494.8, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.176, "pp_tps": 2909.7, "tg_tokens": 128, "tg_s": 2.288, "tg_tps": 55.51, "raw": {"pp": [0.18308231001719832, 0.17596286674961448, 0.21423810021951795], "tg": [2.3049342604354024, 2.3430285179056227, 2.3147874921560287], "tg1": [0.017149847000837326, 0.016962516121566296, 0.019108522683382034]}} +{"tag": "leader-nosplitkv", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 78.64, "rss_mb": 8450.2, "rss_peak_mb": 8473.2, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1661, "pp_tps": 3082.72, "tg_tokens": 128, "tg_s": 2.0036, "tg_tps": 63.39, "raw": {"pp": [0.1672560372389853, 0.1660872590728104, 0.18195243133231997], "tg": [2.0187266608700156, 2.078219319693744, 2.063376694917679], "tg1": [0.016000438015908003, 0.01626650895923376, 0.015106705948710442]}} +{"tag": "leader-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 51.93, "rss_mb": 8471.4, "rss_peak_mb": 8489.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1661, "pp_tps": 3082.69, "tg_tokens": 128, "tg_s": 2.0161, "tg_tps": 62.99, "raw": {"pp": [0.16608853498473763, 0.16695645125582814, 0.17180946189910173], "tg": [2.0667635132558644, 2.0320902089588344, 2.0568131166510284], "tg1": [0.016675901133567095, 0.01703286776319146, 0.015959253069013357]}} +{"tag": "leader", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 53.03, "rss_mb": 8450.2, "rss_peak_mb": 8474.0, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1678, "pp_tps": 3051.95, "tg_tokens": 128, "tg_s": 2.0026, "tg_tps": 63.42, "raw": {"pp": [0.1677613421343267, 0.17518142703920603, 0.17216769512742758], "tg": [2.0181680670939386, 2.045619810000062, 2.017581162042916], "tg1": [0.014953931327909231, 0.015587549190968275, 0.01536224177107215]}} +{"tag": "leader-nosplitkv", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 52.42, "rss_mb": 8449.9, "rss_peak_mb": 8474.8, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1495, "pp_tps": 3425.09, "tg_tokens": 128, "tg_s": 1.7717, "tg_tps": 71.68, "raw": {"pp": [0.1503864680416882, 0.1627977816388011, 0.1494850511662662], "tg": [1.7899064039811492, 1.7877749968320131, 1.7861353252083063], "tg1": [0.01460887212306261, 0.014465874060988426, 0.014931404031813145]}} +{"tag": "leader-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 71.6, "rss_mb": 8448.7, "rss_peak_mb": 8472.9, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.6363, "pp_tps": 804.61, "tg_tokens": 128, "tg_s": 3.9901, "tg_tps": 31.83, "raw": {"pp": [0.6363332280889153, 0.6899973549880087, 0.9538670806214213], "tg": [10.062090303748846, 11.140456527005881, 4.0063140746206045], "tg1": [0.01683283317834139, 0.01622168067842722, 0.01864417502656579]}} +{"tag": "leader", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.31, "rss_mb": 8456.8, "rss_peak_mb": 8480.4, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1517, "pp_tps": 3374.74, "tg_tokens": 128, "tg_s": 1.7671, "tg_tps": 71.87, "raw": {"pp": [0.15171556919813156, 0.15706020314246416, 0.154549116268754], "tg": [1.782004885841161, 1.8077111751772463, 1.8367386371828616], "tg1": [0.017101087141782045, 0.014878218993544579, 0.015195842832326889]}} +{"tag": "leader-nosplitkv", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 51.44, "rss_mb": 8442.5, "rss_peak_mb": 8467.3, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.154, "pp_tps": 3324.69, "tg_tokens": 128, "tg_s": 1.7687, "tg_tps": 71.8, "raw": {"pp": [0.15424893703311682, 0.15522084012627602, 0.15399918612092733], "tg": [1.8099028407596052, 1.8291758550330997, 1.7837468180805445], "tg1": [0.015516853891313076, 0.015281836967915297, 0.015004676301032305]}} +{"tag": "leader-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 50.8, "rss_mb": 8449.5, "rss_peak_mb": 8473.7, "threads_bind": "32-55", "omp_num_threads": "24", "pp_tokens": 512, "pp_s": 0.1511, "pp_tps": 3389.54, "tg_tokens": 128, "tg_s": 1.7391, "tg_tps": 73.03, "raw": {"pp": [0.1712735160253942, 0.15105291502550244, 0.15270386030897498], "tg": [1.7530193431302905, 1.7823738330043852, 1.7983860420063138], "tg1": [0.014007281046360731, 0.014001580886542797, 0.013904284220188856]}} diff --git a/benchmarks/edge_harness/results/parity_cpu.json b/benchmarks/edge_harness/results/parity_cpu.json new file mode 100644 index 000000000..f3fb4052d --- /dev/null +++ b/benchmarks/edge_harness/results/parity_cpu.json @@ -0,0 +1,13 @@ +{ + "ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single", + "vllm_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single", + "greedy_token_match": "24/24", + "exact_match": true, + "ref_top1": 3066, + "vllm_first_token": 3066, + "first_token_agree": true, + "top20_logprob_overlap": "20/20", + "top20_logprob_max_abs_diff": 0.23139667510986328, + "top20_logprob_mean_abs_diff": 0.09972160011529922, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int4_g128_long.json b/benchmarks/edge_harness/results/parity_int4_g128_long.json new file mode 100644 index 000000000..7792886e8 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int4_g128_long.json @@ -0,0 +1,13 @@ +{ + "ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district", + "vllm_text": "We are given a list of items with codes for warehouse districts from 1 to 120. The list is:\n\nItem 1: ", + "greedy_token_match": "7/32", + "exact_match": false, + "ref_top1": 3066, + "vllm_first_token": 3066, + "first_token_agree": true, + "top20_logprob_overlap": "15/20", + "top20_logprob_max_abs_diff": 5.392128944396973, + "top20_logprob_mean_abs_diff": 1.7514413834083826, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int4_g128_short.json b/benchmarks/edge_harness/results/parity_int4_g128_short.json new file mode 100644 index 000000000..6cdcc9f64 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int4_g128_short.json @@ -0,0 +1,13 @@ +{ + "ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single", + "vllm_text": "We need to answer: \"What is the capital of France? Answer in one word.\" The capital of France is Paris", + "greedy_token_match": "1/24", + "exact_match": false, + "ref_top1": 3066, + "vllm_first_token": 3066, + "first_token_agree": true, + "top20_logprob_overlap": "16/20", + "top20_logprob_max_abs_diff": 5.293044328689575, + "top20_logprob_mean_abs_diff": 1.9501573741436005, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int4_g128_tools.json b/benchmarks/edge_harness/results/parity_int4_g128_tools.json new file mode 100644 index 000000000..5317524bb --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int4_g128_tools.json @@ -0,0 +1,13 @@ +{ + "ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current", + "vllm_text": "The user wants to know the weather in Shenzhen right now in celsius. I have a function available called `get_weather` that can get weather", + "greedy_token_match": "29/32", + "exact_match": false, + "ref_top1": 1046, + "vllm_first_token": 1046, + "first_token_agree": true, + "top20_logprob_overlap": "9/20", + "top20_logprob_max_abs_diff": 10.999816656112671, + "top20_logprob_mean_abs_diff": 6.1318019396728936, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int4_g64_long.json b/benchmarks/edge_harness/results/parity_int4_g64_long.json new file mode 100644 index 000000000..3b3e79e30 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int4_g64_long.json @@ -0,0 +1,13 @@ +{ + "ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district", + "vllm_text": "Got it, let's see. The user is asking for the code of warehouse district 73. Let's look through the inventory list. Let's check", + "greedy_token_match": "0/32", + "exact_match": false, + "ref_top1": 3066, + "vllm_first_token": 71430, + "first_token_agree": false, + "top20_logprob_overlap": "12/20", + "top20_logprob_max_abs_diff": 5.217943981289864, + "top20_logprob_mean_abs_diff": 2.0544758004446826, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int4_g64_short.json b/benchmarks/edge_harness/results/parity_int4_g64_short.json new file mode 100644 index 000000000..d3ae47ad1 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int4_g64_short.json @@ -0,0 +1,13 @@ +{ + "ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single", + "vllm_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be just \"", + "greedy_token_match": "22/24", + "exact_match": false, + "ref_top1": 3066, + "vllm_first_token": 3066, + "first_token_agree": true, + "top20_logprob_overlap": "16/20", + "top20_logprob_max_abs_diff": 3.336427927017212, + "top20_logprob_mean_abs_diff": 1.2069085873663425, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int4_g64_tools.json b/benchmarks/edge_harness/results/parity_int4_g64_tools.json new file mode 100644 index 000000000..7a52aeac3 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int4_g64_tools.json @@ -0,0 +1,13 @@ +{ + "ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current", + "vllm_text": "The user wants to know the weather in Shenzhen right now, in celsius. I need to use the get_weather function to fetch the current weather", + "greedy_token_match": "13/32", + "exact_match": false, + "ref_top1": 1046, + "vllm_first_token": 1046, + "first_token_agree": true, + "top20_logprob_overlap": "11/20", + "top20_logprob_max_abs_diff": 5.049500465393066, + "top20_logprob_mean_abs_diff": 2.8302731025456027, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int8.json b/benchmarks/edge_harness/results/parity_int8.json new file mode 100644 index 000000000..39e418d1d --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int8.json @@ -0,0 +1,13 @@ +{ + "ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single", + "vllm_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be a single", + "greedy_token_match": "23/24", + "exact_match": false, + "ref_top1": 3066, + "vllm_first_token": 3066, + "first_token_agree": true, + "top20_logprob_overlap": "18/20", + "top20_logprob_max_abs_diff": 0.8292734622955322, + "top20_logprob_mean_abs_diff": 0.3445668758617507, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int8_long.json b/benchmarks/edge_harness/results/parity_int8_long.json new file mode 100644 index 000000000..bb55f4726 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int8_long.json @@ -0,0 +1,13 @@ +{ + "ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district", + "vllm_text": "We are given an inventory listing with item codes for warehouse districts. The question asks: \"What is the code for warehouse district 73?\" We need to", + "greedy_token_match": "12/32", + "exact_match": false, + "ref_top1": 3066, + "vllm_first_token": 3066, + "first_token_agree": true, + "top20_logprob_overlap": "19/20", + "top20_logprob_max_abs_diff": 1.5648207664489746, + "top20_logprob_mean_abs_diff": 0.7438137785678631, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_int8_tools.json b/benchmarks/edge_harness/results/parity_int8_tools.json new file mode 100644 index 000000000..822730740 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_int8_tools.json @@ -0,0 +1,13 @@ +{ + "ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current", + "vllm_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve weather", + "greedy_token_match": "31/32", + "exact_match": false, + "ref_top1": 1046, + "vllm_first_token": 1046, + "first_token_agree": true, + "top20_logprob_overlap": "18/20", + "top20_logprob_max_abs_diff": 0.5839385986328125, + "top20_logprob_mean_abs_diff": 0.21885341487520943, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_llamacpp_q4km_long.json b/benchmarks/edge_harness/results/parity_llamacpp_q4km_long.json new file mode 100644 index 000000000..d134db4e3 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_llamacpp_q4km_long.json @@ -0,0 +1,14 @@ +{ + "engine": "llama.cpp", + "gguf": "Spark-X2.5-1.7B-Q4_K_M.gguf", + "ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district", + "llamacpp_text": "We are given an inventory listing with item 1 to item 120, each with a code for a warehouse district. The codes are given as:", + "greedy_token_match": "10/32", + "exact_match": false, + "ref_top1": 3066, + "llamacpp_first_token": 3066, + "first_token_agree": true, + "top20_prob_overlap": "16/20", + "top20_prob_max_abs_diff": 0.0041646957494917425, + "top20_prob_mean_abs_diff": 0.0005320748771736946 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_llamacpp_q4km_short.json b/benchmarks/edge_harness/results/parity_llamacpp_q4km_short.json new file mode 100644 index 000000000..5419d3462 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_llamacpp_q4km_short.json @@ -0,0 +1,14 @@ +{ + "engine": "llama.cpp", + "gguf": "Spark-X2.5-1.7B-Q4_K_M.gguf", + "ref_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single", + "llamacpp_text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be a single", + "greedy_token_match": "23/24", + "exact_match": false, + "ref_top1": 3066, + "llamacpp_first_token": 3066, + "first_token_agree": true, + "top20_prob_overlap": "14/20", + "top20_prob_max_abs_diff": 0.21300125134717218, + "top20_prob_mean_abs_diff": 0.030018658851745834 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_llamacpp_q4km_tools.json b/benchmarks/edge_harness/results/parity_llamacpp_q4km_tools.json new file mode 100644 index 000000000..3255b03b7 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_llamacpp_q4km_tools.json @@ -0,0 +1,14 @@ +{ + "engine": "llama.cpp", + "gguf": "Spark-X2.5-1.7B-Q4_K_M.gguf", + "ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current", + "llamacpp_text": "The user wants weather for Shenzhen in celsius. I have the weather function available. Let me call it with the city \"Shenzhen\"", + "greedy_token_match": "3/32", + "exact_match": false, + "ref_top1": 1046, + "llamacpp_first_token": 1046, + "first_token_agree": true, + "top20_prob_overlap": "13/20", + "top20_prob_max_abs_diff": 0.0003068447281868725, + "top20_prob_mean_abs_diff": 5.7490022653347724e-05 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_long.json b/benchmarks/edge_harness/results/parity_long.json new file mode 100644 index 000000000..e5c43ed88 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_long.json @@ -0,0 +1,13 @@ +{ + "ref_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district", + "vllm_text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district", + "greedy_token_match": "32/32", + "exact_match": true, + "ref_top1": 3066, + "vllm_first_token": 3066, + "first_token_agree": true, + "top20_logprob_overlap": "20/20", + "top20_logprob_max_abs_diff": 0.2409982681274414, + "top20_logprob_mean_abs_diff": 0.146433162339963, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_tools.json b/benchmarks/edge_harness/results/parity_tools.json new file mode 100644 index 000000000..abbc8cb40 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_tools.json @@ -0,0 +1,13 @@ +{ + "ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current", + "vllm_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available to get weather for a city. Let me call", + "greedy_token_match": "22/32", + "exact_match": false, + "ref_top1": 1046, + "vllm_first_token": 1046, + "first_token_agree": true, + "top20_logprob_overlap": "19/20", + "top20_logprob_max_abs_diff": 0.4140052795410156, + "top20_logprob_mean_abs_diff": 0.21458697107032335, + "dtype": "bfloat16" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/parity_tools_fp32.json b/benchmarks/edge_harness/results/parity_tools_fp32.json new file mode 100644 index 000000000..6965d47e1 --- /dev/null +++ b/benchmarks/edge_harness/results/parity_tools_fp32.json @@ -0,0 +1,13 @@ +{ + "ref_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve weather", + "vllm_text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve weather", + "greedy_token_match": "32/32", + "exact_match": true, + "ref_top1": 1046, + "vllm_first_token": 1046, + "first_token_agree": true, + "top20_logprob_overlap": "20/20", + "top20_logprob_max_abs_diff": 2.193450927734375e-05, + "top20_logprob_mean_abs_diff": 1.334088810835965e-05, + "dtype": "float32" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/ppl_vllm.jsonl b/benchmarks/edge_harness/results/ppl_vllm.jsonl new file mode 100644 index 000000000..9c8ba4af6 --- /dev/null +++ b/benchmarks/edge_harness/results/ppl_vllm.jsonl @@ -0,0 +1,9 @@ +{"tag": "vllm-bf16", "model": "XHToken/Spark-X2.5-1.7B", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.7763726822277026, "ppl": 43.657394933805286} +{"tag": "vllm-int4-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.871370641182274, "ppl": 48.00814294379576} +{"tag": "vllm-int4-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.949247471796542, "ppl": 51.89629870830851} +{"tag": "vllm-int4-g32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g32", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 4.041093142384169, "ppl": 56.88849605431311} +{"tag": "vllm-int4-g64-fpembed", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64-fpembed", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.884205416004657, "ppl": 48.62828784511256} +{"tag": "vllm-int4-g32-rtn", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g32-rtn", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 4.039075346914289, "ppl": 56.7738224375464} +{"tag": "vllm-int4-g64-rtn", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64-rtn", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 4.089284296531948, "ppl": 59.69715095314776} +{"tag": "expanded-g32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g32-expanded", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 4.039537154311983, "ppl": 56.80004706364053} +{"tag": "expanded-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64-expanded", "window": 512, "chunks": 143, "scored_tokens": 73073, "nll": 3.872951761505493, "ppl": 48.084109634697604} diff --git a/benchmarks/edge_harness/results/ppl_vllm_corpus2.jsonl b/benchmarks/edge_harness/results/ppl_vllm_corpus2.jsonl new file mode 100644 index 000000000..1cbd89ae8 --- /dev/null +++ b/benchmarks/edge_harness/results/ppl_vllm_corpus2.jsonl @@ -0,0 +1,6 @@ +{"tag": "c2-bf16", "model": "XHToken/Spark-X2.5-1.7B", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.3291262563013975, "ppl": 10.268965168486398} +{"tag": "c2-int4-g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.446570275075721, "ppl": 11.548669956733075} +{"tag": "c2-int4-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.4960842284453117, "ppl": 12.134883374312405} +{"tag": "c2-int4-g32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g32", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.4446298061138063, "ppl": 11.526281849868935} +{"tag": "c2-gptq-w4a8", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.5695190991422434, "ppl": 13.05954258534955} +{"tag": "c2-gptq-w4a16", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "window": 512, "chunks": 247, "scored_tokens": 126217, "nll": 2.5562905847375963, "ppl": 12.887921884302088} diff --git a/benchmarks/edge_harness/results/profile_block4_fp16.json b/benchmarks/edge_harness/results/profile_block4_fp16.json new file mode 100644 index 000000000..605f4e461 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_block4_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 6103, "estimated_inference_peak_memory": 154943488, "first_load_time": 173117, "first_load_peak_memory": 113643520, "warm_load_time": 176691, "warm_load_peak_memory": 124940288, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2398992], "first_load_memory_peak_range": [1228800, 10888560], "warm_load_memory_increase_range": [0, 2296256], "warm_load_memory_peak_range": [335872, 10494528], "inference_memory_increase_range": [0, 6340656], "inference_memory_peak_range": [6332416, 15690592], "all_compile_times": [], "all_first_load_times": [173117], "all_warm_load_times": [176691], "all_inference_times": [8046, 6381, 6308, 6330, 6422, 6444, 6621, 6554, 6463, 7236, 6207, 6835, 6499, 6191, 6175, 6205, 6177, 6197, 6160, 6560, 6171, 6167, 6156, 6177, 6149, 6226, 6159, 6146, 6144, 6144, 6362, 6186, 6130, 6141, 6193, 6135, 6392, 6330, 6145, 6146, 6141, 6123, 6151, 6191, 7266, 6796, 6240, 6180, 6170, 6179, 6157, 6303, 6285, 6554, 6221, 6585, 6215, 6153, 6157, 6133, 6271, 6167, 6194, 6285, 6137, 6142, 6153, 6253, 6159, 6185, 6195, 6229, 7027, 6126, 6135, 6129, 6135, 6155, 6499, 6235, 6134, 6145, 6112, 6214, 6160, 6168, 6912, 6158, 6122, 6273, 6404, 6111, 6152, 6135, 6123, 6121, 6107, 6123, 6103, 6181]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 160867}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 135879}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10234}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_399", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 252656}, {"name": "node_Slice_402", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 59233}, {"name": "node_Slice_405", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 62582}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 495}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 249}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 267}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7496}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6813}, {"name": "node_Slice_1001", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5076}, {"name": "node_Slice_1004", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 23170}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1040}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1180}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 340}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 963}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17162}, {"name": "node_Slice_2566", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_2569", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3367}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18517}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 344}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5014}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 225}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 116401}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 456203}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9395}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11870}, {"name": "/layers.0/ReduceSum", "type": "/layers.0/ReduceSum", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12926}, {"name": "/layers.0/Mul_8", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4368}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1209}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 96928}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22165}, {"name": "node_Slice_1202", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_1205", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9717}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 105429}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 36403}, {"name": "/layers.0/Mul_9", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3890}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5853}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 469}, {"name": "/layers.0/Mul_10", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3245}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 152121}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_5", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 70561}, {"name": "rms_norm_/layers.0/Mul_12_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4470}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 804262}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 788118}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_15", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11671}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 791431}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_8", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6921}, {"name": "rms_norm_/layers.1/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3944}, {"name": "/layers.1/q_k_v_proj/MatMul", "type": "/layers.1/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/q_k_v_proj/MatMul_post_reshape", "type": "/layers.1/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_409", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 251811}, {"name": "node_Slice_412", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 60914}, {"name": "node_Slice_415", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 65435}, {"name": "/layers.1/Reshape", "type": "/layers.1/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 526}, {"name": "/layers.1/Reshape_1", "type": "/layers.1/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 243}, {"name": "/layers.1/Reshape_2", "type": "/layers.1/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 245}, {"name": "/layers.1/g_proj/MatMul", "type": "/layers.1/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/g_proj/MatMul_post_reshape", "type": "/layers.1/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Sigmoid", "type": "/layers.1/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4023}, {"name": "/layers.1/Reshape_3", "type": "/layers.1/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 851}, {"name": "node_Slice_2573", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_2576", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Mul_2", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1048}, {"name": "/layers.1/Neg", "type": "/layers.1/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1204}, {"name": "/layers.1/Concat", "type": "/layers.1/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Mul_3", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 984}, {"name": "/layers.1/Add_1", "type": "/layers.1/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5316}, {"name": "node_Slice_2580", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_2583", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Mul_4", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 398}, {"name": "/layers.1/Neg_1", "type": "/layers.1/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1112}, {"name": "/layers.1/Concat_1", "type": "/layers.1/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Mul_5", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 363}, {"name": "/layers.1/Add_2", "type": "/layers.1/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 941}, {"name": "/layers.1/Reshape_4", "type": "/layers.1/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 211}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_1_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 107571}, {"name": "/layers.1/Transpose", "type": "/layers.1/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/MatMul", "type": "/layers.1/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 248512}, {"name": "/layers.1/Mul_6", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6852}, {"name": "/layers.1/Mul_7", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4133}, {"name": "/layers.1/ReduceSum", "type": "/layers.1/ReduceSum", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2850}, {"name": "/layers.1/Mul_8", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1773}, {"name": "/layers.1/Concat_2", "type": "/layers.1/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Add_3", "type": "/layers.1/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12370}, {"name": "/layers.1/Softmax", "type": "/layers.1/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5507}, {"name": "node_Slice_1036", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_1039", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4464}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_1_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 105004}, {"name": "/layers.1/MatMul_1", "type": "/layers.1/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 42048}, {"name": "/layers.1/Mul_9", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4523}, {"name": "/layers.1/Add_4", "type": "/layers.1/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4362}, {"name": "/layers.1/Reshape_5", "type": "/layers.1/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 437}, {"name": "/layers.1/Mul_10", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 912}, {"name": "/layers.1/out_proj/MatMul", "type": "/layers.1/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 169538}, {"name": "/layers.1/out_proj/MatMul_post_reshape", "type": "/layers.1/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Add_5", "type": "/layers.1/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6735}, {"name": "rms_norm_/layers.1/Mul_12_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4174}, {"name": "/layers.1/gate_proj/MatMul", "type": "/layers.1/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/gate_proj/MatMul_post_reshape", "type": "/layers.1/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_1", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 784833}, {"name": "/layers.1/up_proj/MatMul", "type": "/layers.1/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 779331}, {"name": "/layers.1/up_proj/MatMul_post_reshape", "type": "/layers.1/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Mul_15", "type": "/layers.1/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11480}, {"name": "/layers.1/down_proj/MatMul", "type": "/layers.1/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 807496}, {"name": "/layers.1/down_proj/MatMul_post_reshape", "type": "/layers.1/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Add_8", "type": "/layers.1/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6734}, {"name": "rms_norm_/layers.2/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5632}, {"name": "/layers.2/q_k_v_proj/MatMul", "type": "/layers.2/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/q_k_v_proj/MatMul_post_reshape", "type": "/layers.2/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_1325", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 250735}, {"name": "node_Slice_1328", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 61151}, {"name": "node_Slice_1331", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 66637}, {"name": "/layers.2/Reshape", "type": "/layers.2/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 436}, {"name": "/layers.2/Reshape_1", "type": "/layers.2/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 219}, {"name": "/layers.2/Reshape_2", "type": "/layers.2/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 215}, {"name": "/layers.2/g_proj/MatMul", "type": "/layers.2/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/g_proj/MatMul_post_reshape", "type": "/layers.2/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Sigmoid", "type": "/layers.2/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4063}, {"name": "/layers.2/Reshape_3", "type": "/layers.2/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 795}, {"name": "node_Slice_357", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_360", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Mul_2", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1009}, {"name": "/layers.2/Neg", "type": "/layers.2/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1089}, {"name": "/layers.2/Concat", "type": "/layers.2/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Mul_3", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 972}, {"name": "/layers.2/Add_1", "type": "/layers.2/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5694}, {"name": "node_Slice_364", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_367", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Mul_4", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 384}, {"name": "/layers.2/Neg_1", "type": "/layers.2/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1117}, {"name": "/layers.2/Concat_1", "type": "/layers.2/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Mul_5", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 393}, {"name": "/layers.2/Add_2", "type": "/layers.2/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 878}, {"name": "/layers.2/Reshape_4", "type": "/layers.2/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 262}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_2_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 92625}, {"name": "/layers.2/Transpose", "type": "/layers.2/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/MatMul", "type": "/layers.2/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 243893}, {"name": "/layers.2/Mul_6", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11304}, {"name": "/layers.2/Mul_7", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4767}, {"name": "/layers.2/ReduceSum", "type": "/layers.2/ReduceSum", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4344}, {"name": "/layers.2/Mul_8", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1296}, {"name": "/layers.2/Concat_2", "type": "/layers.2/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Add_3", "type": "/layers.2/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11414}, {"name": "/layers.2/Softmax", "type": "/layers.2/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5029}, {"name": "node_Slice_371", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_374", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5563}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_2_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 117744}, {"name": "/layers.2/MatMul_1", "type": "/layers.2/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35549}, {"name": "/layers.2/Mul_9", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3757}, {"name": "/layers.2/Add_4", "type": "/layers.2/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3972}, {"name": "/layers.2/Reshape_5", "type": "/layers.2/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 444}, {"name": "/layers.2/Mul_10", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1387}, {"name": "/layers.2/out_proj/MatMul", "type": "/layers.2/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 211582}, {"name": "/layers.2/out_proj/MatMul_post_reshape", "type": "/layers.2/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Add_5", "type": "/layers.2/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7735}, {"name": "rms_norm_/layers.2/Mul_12_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3612}, {"name": "/layers.2/gate_proj/MatMul", "type": "/layers.2/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/gate_proj/MatMul_post_reshape", "type": "/layers.2/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_2", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 858318}, {"name": "/layers.2/up_proj/MatMul", "type": "/layers.2/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 796230}, {"name": "/layers.2/up_proj/MatMul_post_reshape", "type": "/layers.2/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Mul_15", "type": "/layers.2/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13558}, {"name": "/layers.2/down_proj/MatMul", "type": "/layers.2/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 780322}, {"name": "/layers.2/down_proj/MatMul_post_reshape", "type": "/layers.2/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Add_8", "type": "/layers.2/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6983}, {"name": "rms_norm_/layers.3/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3544}, {"name": "/layers.3/q_k_v_proj/MatMul", "type": "/layers.3/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/q_k_v_proj/MatMul_post_reshape", "type": "/layers.3/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_1335", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 252410}, {"name": "node_Slice_1338", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 63173}, {"name": "node_Slice_1341", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 59786}, {"name": "/layers.3/Reshape", "type": "/layers.3/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 514}, {"name": "/layers.3/Reshape_1", "type": "/layers.3/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 274}, {"name": "/layers.3/Reshape_2", "type": "/layers.3/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 214}, {"name": "/layers.3/g_proj/MatMul", "type": "/layers.3/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/g_proj/MatMul_post_reshape", "type": "/layers.3/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Sigmoid", "type": "/layers.3/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4866}, {"name": "/layers.3/Reshape_3", "type": "/layers.3/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1917}, {"name": "node_Slice_378", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_381", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 763}, {"name": "fused_/layers.3/Slice_3_/layers.3/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.3/Slice_3_/layers.3/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 403}, {"name": "/layers.3/Mul_2", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 360}, {"name": "/layers.3/Neg", "type": "/layers.3/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 439}, {"name": "/layers.3/Concat", "type": "/layers.3/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 366}, {"name": "/layers.3/Mul_3", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 300}, {"name": "/layers.3/Add_1", "type": "/layers.3/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 878}, {"name": "/layers.3/Concat_1", "type": "/layers.3/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2381}, {"name": "node_Slice_385", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_388", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.3/Slice_7_/layers.3/Slice_9", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.3/Slice_7_/layers.3/Slice_10", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Mul_4", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 225}, {"name": "/layers.3/Neg_1", "type": "/layers.3/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 597}, {"name": "/layers.3/Concat_2", "type": "/layers.3/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Mul_5", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 201}, {"name": "/layers.3/Add_2", "type": "/layers.3/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 217}, {"name": "/layers.3/Concat_3", "type": "/layers.3/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Reshape_4", "type": "/layers.3/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 208}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_3_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 138545}, {"name": "/layers.3/Transpose", "type": "/layers.3/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/MatMul", "type": "/layers.3/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 450217}, {"name": "/layers.3/Mul_6", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14017}, {"name": "/layers.3/Mul_7", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3134}, {"name": "/layers.3/ReduceSum", "type": "/layers.3/ReduceSum", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1648}, {"name": "/layers.3/Mul_8", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 534}, {"name": "/layers.3/Concat_4", "type": "/layers.3/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6297}, {"name": "/layers.3/Add_3", "type": "/layers.3/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20477}, {"name": "/layers.3/Softmax", "type": "/layers.3/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6507}, {"name": "node_Slice_392", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_395", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_3_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 170588}, {"name": "/layers.3/MatMul_1", "type": "/layers.3/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 170262}, {"name": "/layers.3/Mul_9", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3684}, {"name": "/layers.3/Add_4", "type": "/layers.3/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7606}, {"name": "/layers.3/Reshape_5", "type": "/layers.3/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 441}, {"name": "/layers.3/Mul_10", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 924}, {"name": "/layers.3/out_proj/MatMul", "type": "/layers.3/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 193264}, {"name": "/layers.3/out_proj/MatMul_post_reshape", "type": "/layers.3/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Add_5", "type": "/layers.3/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6918}, {"name": "rms_norm_/layers.3/Mul_12_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3566}, {"name": "/layers.3/gate_proj/MatMul", "type": "/layers.3/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/gate_proj/MatMul_post_reshape", "type": "/layers.3/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_3", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 792873}, {"name": "/layers.3/up_proj/MatMul", "type": "/layers.3/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 780398}, {"name": "/layers.3/up_proj/MatMul_post_reshape", "type": "/layers.3/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Mul_15", "type": "/layers.3/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16344}, {"name": "/layers.3/down_proj/MatMul", "type": "/layers.3/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 789082}, {"name": "/layers.3/down_proj/MatMul_post_reshape", "type": "/layers.3/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Add_8", "type": "/layers.3/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6894}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Add_2_identity", "type": "/layers.1/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.1/Reshape_2_identity", "type": "/layers.1/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Add_2_identity", "type": "/layers.2/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.2/Reshape_2_identity", "type": "/layers.2/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Concat_3_identity", "type": "/layers.3/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.3/Reshape_2_identity", "type": "/layers.3/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_3_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_4_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_5_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_6_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_7_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_8_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 48049}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_calib_w8a16.json b/benchmarks/edge_harness/results/profile_calib_w8a16.json new file mode 100644 index 000000000..f8abe3210 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_calib_w8a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 803, "estimated_inference_peak_memory": 138977280, "first_load_time": 104796, "first_load_peak_memory": 113455104, "warm_load_time": 105370, "warm_load_peak_memory": 105713664, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [3270000, 12452832], "first_load_memory_peak_range": [11902976, 21085808], "warm_load_memory_increase_range": [26110560, 36053536], "warm_load_memory_peak_range": [15630336, 25573312], "inference_memory_increase_range": [0, 1065008], "inference_memory_peak_range": [0, 9549376], "all_compile_times": [], "all_first_load_times": [104796], "all_warm_load_times": [105370], "all_inference_times": [1533, 870, 840, 852, 838, 813, 812, 808, 811, 814, 821, 973, 861, 843, 852, 837, 835, 843, 833, 824, 807, 808, 815, 812, 816, 822, 837, 835, 815, 807, 815, 808, 815, 806, 805, 805, 806, 857, 826, 805, 805, 804, 804, 810, 803, 804, 806, 804, 816, 805, 806, 811, 812, 818, 810, 805, 806, 806, 818, 807, 805, 806, 806, 819, 807, 806, 807, 848, 815, 1201, 987, 806, 805, 813, 805, 805, 805, 810, 804, 807, 804, 805, 806, 805, 813, 804, 810, 866, 837, 819, 803, 803, 806, 804, 812, 805, 803, 806, 804, 815]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 38815}, {"name": "/layers.0/Pow", "type": "/layers.0/Pow", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 54240}, {"name": "/layers.0/ReduceMean", "type": "/layers.0/ReduceMean", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10302}, {"name": "/layers.0/Add", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11317}, {"name": "/layers.0/Sqrt", "type": "/layers.0/Sqrt", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Div", "type": "/layers.0/Div", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 72747}, {"name": "/layers.0/Mul", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 87612}, {"name": "/layers.0/Mul_1", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 28026}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10746}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_647", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 100071}, {"name": "node_Slice_650", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20997}, {"name": "node_Slice_653", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7438}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 302}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6351}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16354}, {"name": "node_Slice_1165", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8720}, {"name": "node_Slice_1168", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1829}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16759}, {"name": "node_Slice_1388", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2646}, {"name": "node_Slice_1391", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8958}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22033}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2967}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11087}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1292}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11157}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14091}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3547}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1063}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 248}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17585}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 197582}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2115}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 37708}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16817}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 59003}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 21553}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 43593}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7006}, {"name": "/layers.0/Pow_1", "type": "/layers.0/Pow", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11195}, {"name": "/layers.0/ReduceMean_1", "type": "/layers.0/ReduceMean", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1481}, {"name": "/layers.0/Add_5", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 186}, {"name": "/layers.0/Sqrt_1", "type": "/layers.0/Sqrt", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Div_1", "type": "/layers.0/Div", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 58619}, {"name": "/layers.0/Mul_8", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 37518}, {"name": "/layers.0/Mul_9", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12218}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 456167}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "_elementwiseneuron_0", "type": "elementwiseneuron", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 427975}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19765}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 432305}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5846}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9779}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_calib_w8a8.json b/benchmarks/edge_harness/results/profile_calib_w8a8.json new file mode 100644 index 000000000..827a62db5 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_calib_w8a8.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 749, "estimated_inference_peak_memory": 138420224, "first_load_time": 111059, "first_load_peak_memory": 113651712, "warm_load_time": 106186, "warm_load_peak_memory": 126267392, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2499648], "first_load_memory_peak_range": [1368064, 11294720], "warm_load_memory_increase_range": [0, 1585792], "warm_load_memory_peak_range": [1028096, 10936192], "inference_memory_increase_range": [0, 544816], "inference_memory_peak_range": [0, 9582304], "all_compile_times": [], "all_first_load_times": [111059], "all_warm_load_times": [106186], "all_inference_times": [1389, 831, 800, 794, 804, 876, 923, 996, 828, 813, 892, 871, 794, 777, 768, 781, 810, 941, 785, 896, 871, 837, 822, 809, 798, 799, 834, 759, 761, 761, 761, 767, 774, 764, 763, 798, 804, 772, 760, 749, 768, 800, 795, 769, 765, 837, 781, 789, 768, 771, 869, 851, 806, 874, 808, 810, 825, 806, 799, 797, 806, 979, 949, 909, 1089, 929, 816, 1141, 965, 782, 828, 811, 824, 834, 842, 870, 849, 826, 854, 844, 833, 821, 911, 964, 1004, 936, 927, 984, 823, 800, 789, 790, 800, 802, 792, 809, 781, 828, 809, 790]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20618}, {"name": "/layers.0/Pow", "type": "/layers.0/Pow", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 50076}, {"name": "/layers.0/ReduceMean", "type": "/layers.0/ReduceMean", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3767}, {"name": "/layers.0/Add", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sqrt", "type": "/layers.0/Sqrt", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Div", "type": "/layers.0/Div", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 264}, {"name": "/layers.0/Mul", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 28389}, {"name": "/layers.0/Mul_1", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22514}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6739}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_1544", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 104968}, {"name": "node_Slice_1547", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12500}, {"name": "node_Slice_1550", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11968}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 183}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 413}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7419}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3771}, {"name": "node_Slice_1583", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1692}, {"name": "node_Slice_1586", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 586}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 658}, {"name": "node_Slice_760", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2431}, {"name": "node_Slice_763", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4262}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9041}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1201}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2316}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1297}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1464}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1314}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2103}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2684}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 368}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 25773}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 92247}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7715}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 15119}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6465}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 38968}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19431}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 66748}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3016}, {"name": "/layers.0/Pow_1", "type": "/layers.0/Pow", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5479}, {"name": "/layers.0/ReduceMean_1", "type": "/layers.0/ReduceMean", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1071}, {"name": "/layers.0/Add_5", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sqrt_1", "type": "/layers.0/Sqrt", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Div_1", "type": "/layers.0/Div", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 188}, {"name": "/layers.0/Mul_8", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 25932}, {"name": "/layers.0/Mul_9", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14808}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 396598}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 436206}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "_elementwiseneuron_0", "type": "elementwiseneuron", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8771}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 34873}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 407323}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1090}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9782}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_gate_0.json b/benchmarks/edge_harness/results/profile_gate_0.json new file mode 100644 index 000000000..0d4f106cc --- /dev/null +++ b/benchmarks/edge_harness/results/profile_gate_0.json @@ -0,0 +1,224 @@ +{ + "tag": "bfloat16/None/eager=False", + "wall_step_ms": 17.25, + "tps": 57.97, + "profiled_total_ms_per_step": 16.484, + "buckets_ms_per_step": { + "gemm": 10.661, + "framework": 3.502, + "other_op": 1.165, + "attn": 1.157 + }, + "top": [ + { + "op": "aten::_weight_int4pack_mm_for_cpu", + "bucket": "gemm", + "ms_per_step": 8.8694, + "calls_per_step": 113.0 + }, + { + "op": "## Call CompiledFxGraph None ##", + "bucket": "framework", + "ms_per_step": 2.7736, + "calls_per_step": 1.0 + }, + { + "op": "aten::mm", + "bucket": "gemm", + "ms_per_step": 1.1547, + "calls_per_step": 28.0 + }, + { + "op": "vllm::cpu_int4_linear", + "bucket": "gemm", + "ms_per_step": 0.6367, + "calls_per_step": 113.0 + }, + { + "op": "vllm::unified_kv_cache_update", + "bucket": "framework", + "ms_per_step": 0.5477, + "calls_per_step": 28.0 + }, + { + "op": "vllm::unified_attention_with_output", + "bucket": "attn", + "ms_per_step": 0.5346, + "calls_per_step": 28.0 + }, + { + "op": "_C::cpu_attention_with_kv_cache", + "bucket": "attn", + "ms_per_step": 0.4428, + "calls_per_step": 28.0 + }, + { + "op": "aten::argmax", + "bucket": "other_op", + "ms_per_step": 0.2687, + "calls_per_step": 1.0 + }, + { + "op": "_C::cpu_attn_reshape_and_cache", + "bucket": "attn", + "ms_per_step": 0.1792, + "calls_per_step": 28.0 + }, + { + "op": "aten::slice", + "bucket": "other_op", + "ms_per_step": 0.1786, + "calls_per_step": 230.0 + }, + { + "op": "Pregraph bytecode", + "bucket": "framework", + "ms_per_step": 0.1727, + "calls_per_step": 1.0 + }, + { + "op": "aten::narrow", + "bucket": "other_op", + "ms_per_step": 0.1042, + "calls_per_step": 112.0 + }, + { + "op": "aten::as_strided", + "bucket": "other_op", + "ms_per_step": 0.0839, + "calls_per_step": 234.0 + }, + { + "op": "aten::copy_", + "bucket": "other_op", + "ms_per_step": 0.0833, + "calls_per_step": 19.0 + }, + { + "op": "aten::split", + "bucket": "other_op", + "ms_per_step": 0.0712, + "calls_per_step": 56.0 + }, + { + "op": "aten::chunk", + "bucket": "other_op", + "ms_per_step": 0.07, + "calls_per_step": 56.0 + }, + { + "op": "aten::view", + "bucket": "other_op", + "ms_per_step": 0.0631, + "calls_per_step": 62.0 + }, + { + "op": "aten::empty", + "bucket": "other_op", + "ms_per_step": 0.0551, + "calls_per_step": 145.0 + }, + { + "op": "aten::index", + "bucket": "other_op", + "ms_per_step": 0.0345, + "calls_per_step": 2.0 + }, + { + "op": "_C::compute_slot_mapping_kernel_impl", + "bucket": "other_op", + "ms_per_step": 0.022, + "calls_per_step": 4.0 + }, + { + "op": "aten::new_empty", + "bucket": "other_op", + "ms_per_step": 0.0187, + "calls_per_step": 28.0 + }, + { + "op": "_C::get_scheduler_metadata", + "bucket": "other_op", + "ms_per_step": 0.0184, + "calls_per_step": 4.0 + }, + { + "op": "aten::_to_copy", + "bucket": "other_op", + "ms_per_step": 0.0168, + "calls_per_step": 5.0 + }, + { + "op": "aten::add", + "bucket": "other_op", + "ms_per_step": 0.0115, + "calls_per_step": 3.0 + }, + { + "op": "aten::fill_", + "bucket": "other_op", + "ms_per_step": 0.0112, + "calls_per_step": 12.0 + }, + { + "op": "aten::resolve_conj", + "bucket": "other_op", + "ms_per_step": 0.0096, + "calls_per_step": 59.0 + }, + { + "op": "AOTDispatcher Runtime Wrapper Prologue", + "bucket": "framework", + "ms_per_step": 0.0077, + "calls_per_step": 1.0 + }, + { + "op": "aten::empty_strided", + "bucket": "other_op", + "ms_per_step": 0.0073, + "calls_per_step": 5.0 + }, + { + "op": "aten::to", + "bucket": "other_op", + "ms_per_step": 0.007, + "calls_per_step": 8.0 + }, + { + "op": "aten::sub", + "bucket": "other_op", + "ms_per_step": 0.0063, + "calls_per_step": 1.0 + }, + { + "op": "aten::lt", + "bucket": "other_op", + "ms_per_step": 0.0049, + "calls_per_step": 1.0 + }, + { + "op": "aten::index_select", + "bucket": "other_op", + "ms_per_step": 0.0043, + "calls_per_step": 1.0 + }, + { + "op": "aten::detach", + "bucket": "other_op", + "ms_per_step": 0.0037, + "calls_per_step": 2.0 + }, + { + "op": "aten::reshape", + "bucket": "other_op", + "ms_per_step": 0.0031, + "calls_per_step": 4.0 + }, + { + "op": "aten::alias", + "bucket": "other_op", + "ms_per_step": 0.0026, + "calls_per_step": 1.0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_gate_1.json b/benchmarks/edge_harness/results/profile_gate_1.json new file mode 100644 index 000000000..256ae10b3 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_gate_1.json @@ -0,0 +1,224 @@ +{ + "tag": "bfloat16/None/eager=False", + "wall_step_ms": 14.345, + "tps": 69.71, + "profiled_total_ms_per_step": 13.87, + "buckets_ms_per_step": { + "gemm": 8.263, + "framework": 3.017, + "other_op": 1.581, + "attn": 1.009 + }, + "top": [ + { + "op": "aten::_weight_int4pack_mm_for_cpu", + "bucket": "gemm", + "ms_per_step": 7.7116, + "calls_per_step": 113.0 + }, + { + "op": "## Call CompiledFxGraph None ##", + "bucket": "framework", + "ms_per_step": 2.2035, + "calls_per_step": 1.0 + }, + { + "op": "vllm::cpu_int4_linear", + "bucket": "gemm", + "ms_per_step": 0.5513, + "calls_per_step": 113.0 + }, + { + "op": "vllm::unified_attention_with_output", + "bucket": "attn", + "ms_per_step": 0.4727, + "calls_per_step": 28.0 + }, + { + "op": "vllm::unified_kv_cache_update", + "bucket": "framework", + "ms_per_step": 0.4491, + "calls_per_step": 28.0 + }, + { + "op": "_C::cpu_attention_with_kv_cache", + "bucket": "attn", + "ms_per_step": 0.3842, + "calls_per_step": 28.0 + }, + { + "op": "aten::copy_", + "bucket": "other_op", + "ms_per_step": 0.2688, + "calls_per_step": 75.0 + }, + { + "op": "aten::argmax", + "bucket": "other_op", + "ms_per_step": 0.2479, + "calls_per_step": 1.0 + }, + { + "op": "vllm::spark_head_gate", + "bucket": "framework", + "ms_per_step": 0.2182, + "calls_per_step": 28.0 + }, + { + "op": "_C::cpu_attn_reshape_and_cache", + "bucket": "attn", + "ms_per_step": 0.1523, + "calls_per_step": 28.0 + }, + { + "op": "aten::slice", + "bucket": "other_op", + "ms_per_step": 0.152, + "calls_per_step": 230.0 + }, + { + "op": "Pregraph bytecode", + "bucket": "framework", + "ms_per_step": 0.1399, + "calls_per_step": 1.0 + }, + { + "op": "aten::mul", + "bucket": "other_op", + "ms_per_step": 0.1302, + "calls_per_step": 28.0 + }, + { + "op": "aten::sum", + "bucket": "other_op", + "ms_per_step": 0.0956, + "calls_per_step": 28.0 + }, + { + "op": "aten::as_strided", + "bucket": "other_op", + "ms_per_step": 0.0952, + "calls_per_step": 290.0 + }, + { + "op": "aten::narrow", + "bucket": "other_op", + "ms_per_step": 0.0844, + "calls_per_step": 112.0 + }, + { + "op": "aten::_to_copy", + "bucket": "other_op", + "ms_per_step": 0.0794, + "calls_per_step": 61.0 + }, + { + "op": "aten::chunk", + "bucket": "other_op", + "ms_per_step": 0.0631, + "calls_per_step": 56.0 + }, + { + "op": "aten::split", + "bucket": "other_op", + "ms_per_step": 0.0547, + "calls_per_step": 56.0 + }, + { + "op": "aten::view", + "bucket": "other_op", + "ms_per_step": 0.0531, + "calls_per_step": 62.0 + }, + { + "op": "aten::empty", + "bucket": "other_op", + "ms_per_step": 0.0456, + "calls_per_step": 145.0 + }, + { + "op": "aten::to", + "bucket": "other_op", + "ms_per_step": 0.0335, + "calls_per_step": 64.0 + }, + { + "op": "aten::empty_strided", + "bucket": "other_op", + "ms_per_step": 0.0316, + "calls_per_step": 61.0 + }, + { + "op": "aten::fill_", + "bucket": "other_op", + "ms_per_step": 0.0244, + "calls_per_step": 40.0 + }, + { + "op": "aten::index", + "bucket": "other_op", + "ms_per_step": 0.0217, + "calls_per_step": 2.0 + }, + { + "op": "aten::unsqueeze", + "bucket": "other_op", + "ms_per_step": 0.0184, + "calls_per_step": 29.0 + }, + { + "op": "_C::compute_slot_mapping_kernel_impl", + "bucket": "other_op", + "ms_per_step": 0.018, + "calls_per_step": 4.0 + }, + { + "op": "_C::get_scheduler_metadata", + "bucket": "other_op", + "ms_per_step": 0.0161, + "calls_per_step": 4.0 + }, + { + "op": "aten::new_empty", + "bucket": "other_op", + "ms_per_step": 0.0149, + "calls_per_step": 28.0 + }, + { + "op": "aten::add", + "bucket": "other_op", + "ms_per_step": 0.0095, + "calls_per_step": 3.0 + }, + { + "op": "AOTDispatcher Runtime Wrapper Prologue", + "bucket": "framework", + "ms_per_step": 0.0063, + "calls_per_step": 1.0 + }, + { + "op": "aten::sub", + "bucket": "other_op", + "ms_per_step": 0.005, + "calls_per_step": 1.0 + }, + { + "op": "aten::lt", + "bucket": "other_op", + "ms_per_step": 0.004, + "calls_per_step": 1.0 + }, + { + "op": "aten::index_select", + "bucket": "other_op", + "ms_per_step": 0.0034, + "calls_per_step": 1.0 + }, + { + "op": "aten::reshape", + "bucket": "other_op", + "ms_per_step": 0.0024, + "calls_per_step": 4.0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_lm_head_w4a16.json b/benchmarks/edge_harness/results/profile_lm_head_w4a16.json new file mode 100644 index 000000000..230ac3110 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_lm_head_w4a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 3868, "estimated_inference_peak_memory": 137539584, "first_load_time": 140576, "first_load_peak_memory": 113246208, "warm_load_time": 149076, "warm_load_peak_memory": 125939712, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 1046944], "first_load_memory_peak_range": [753664, 9164752], "warm_load_memory_increase_range": [0, 2845344], "warm_load_memory_peak_range": [2326528, 11678224], "inference_memory_increase_range": [0, 75920], "inference_memory_peak_range": [24576, 8852352], "all_compile_times": [], "all_first_load_times": [140576], "all_warm_load_times": [149076], "all_inference_times": [5046, 4616, 4128, 4077, 4103, 4020, 3940, 3894, 4265, 4737, 4008, 4543, 4147, 4109, 4144, 4048, 3980, 3885, 3903, 3879, 3882, 3901, 3951, 3895, 3926, 3923, 3963, 3895, 3873, 4657, 3889, 3969, 3880, 3892, 4748, 3893, 3895, 3891, 3887, 3876, 3868, 3903, 3879, 3932, 4737, 3915, 3874, 3892, 3889, 3908, 3887, 3898, 3880, 3897, 3877, 3889, 3887, 3890, 3877, 4039, 3941, 3964, 3876, 3892, 3871, 3900, 3887, 3928, 3992, 3956, 3974, 5026, 4694, 4079, 4146, 4183, 4420, 4226, 4090, 4034, 3879, 3956, 3985, 3897, 3879, 3885, 3874, 3906, 3875, 3893, 3881, 4595, 3884, 3885, 3876, 3890, 3877, 3880, 3882, 3875]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11685}, {"name": "rms_norm_/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 86165}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1495}, {"name": "/lm_head/MatMul", "type": "/lm", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9206236}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35181}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_lm_head_w8a16.json b/benchmarks/edge_harness/results/profile_lm_head_w8a16.json new file mode 100644 index 000000000..31854d66c --- /dev/null +++ b/benchmarks/edge_harness/results/profile_lm_head_w8a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 3862, "estimated_inference_peak_memory": 138051584, "first_load_time": 139684, "first_load_peak_memory": 113541120, "warm_load_time": 146509, "warm_load_peak_memory": 125521920, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [2583936, 12322976], "first_load_memory_peak_range": [11759616, 21498656], "warm_load_memory_increase_range": [0, 1511168], "warm_load_memory_peak_range": [983040, 11197440], "inference_memory_increase_range": [0, 24624], "inference_memory_peak_range": [20480, 9138416], "all_compile_times": [], "all_first_load_times": [139684], "all_warm_load_times": [146509], "all_inference_times": [5092, 4632, 4065, 3881, 3887, 3880, 3894, 3879, 3888, 3888, 3876, 4173, 4120, 3950, 3914, 3967, 3883, 3897, 3912, 3978, 3984, 3885, 3951, 3901, 3879, 3948, 3880, 3891, 3880, 3890, 3965, 3891, 3884, 3891, 3897, 3885, 3881, 3882, 3905, 3880, 3895, 3898, 3909, 3862, 3877, 3877, 3872, 3872, 3923, 3928, 4000, 3953, 3980, 3959, 3962, 4915, 4755, 4061, 4097, 4072, 4171, 4131, 4070, 4054, 3944, 3908, 3892, 3887, 3877, 3918, 3982, 3874, 3873, 3874, 3893, 4553, 3871, 3876, 3888, 4111, 3866, 3893, 3870, 3877, 4307, 4161, 4048, 3968, 3883, 3936, 4038, 3969, 3958, 4774, 4849, 4110, 4140, 4342, 4381, 4277]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14198}, {"name": "rms_norm_/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 151909}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1307}, {"name": "/lm_head/MatMul", "type": "/lm", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9681411}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35501}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_ring2_sliding_fp16.json b/benchmarks/edge_harness/results/profile_ring2_sliding_fp16.json new file mode 100644 index 000000000..9f382caa4 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_ring2_sliding_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1480, "estimated_inference_peak_memory": 140103680, "first_load_time": 111241, "first_load_peak_memory": 113639424, "warm_load_time": 115382, "warm_load_peak_memory": 125554688, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2062624], "first_load_memory_peak_range": [1265664, 10942624], "warm_load_memory_increase_range": [0, 4168816], "warm_load_memory_peak_range": [3784704, 13497104], "inference_memory_increase_range": [0, 2121776], "inference_memory_peak_range": [0, 8779808], "all_compile_times": [], "all_first_load_times": [111241], "all_warm_load_times": [115382], "all_inference_times": [2204, 1622, 1592, 1598, 1612, 1603, 1548, 1581, 1571, 1685, 2255, 1731, 1547, 1667, 1673, 1719, 1543, 1534, 1513, 1505, 1495, 1494, 1515, 1498, 1492, 1507, 1503, 1529, 1519, 1490, 1489, 1494, 1487, 1491, 1490, 1483, 1495, 1488, 1490, 1489, 1505, 1496, 1497, 1538, 1518, 1532, 1532, 1525, 1530, 1505, 1519, 1521, 1827, 1747, 1985, 1681, 2027, 1567, 1609, 1607, 1751, 1855, 1866, 1829, 1804, 1615, 1550, 1606, 1556, 1560, 1597, 1576, 2375, 1547, 1522, 1544, 1519, 1483, 1494, 1482, 1482, 1497, 1480, 1485, 1503, 1510, 1486, 1511, 1491, 1481, 1498, 1484, 1562, 1670, 1582, 1666, 1716, 1618, 1711, 1642]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6592}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 107275}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 23416}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_347", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 251480}, {"name": "node_Slice_350", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 60490}, {"name": "node_Slice_353", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 39159}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 482}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 258}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20778}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8519}, {"name": "node_Slice_521", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_524", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22527}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1026}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1024}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2223}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 979}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8600}, {"name": "node_Slice_476", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_479", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4362}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 15065}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 341}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2615}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 250}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 146812}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 315975}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 56548}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6455}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 72308}, {"name": "node_Reshape_2", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 24850}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11607}, {"name": "/layers.0/ReduceMax", "type": "/layers.0/ReduceMax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5899}, {"name": "/layers.0/Max", "type": "/layers.0/Max", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3265}, {"name": "/layers.0/Sub", "type": "/layers.0/Sub", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18352}, {"name": "/layers.0/Exp", "type": "/layers.0/Exp", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6647}, {"name": "/layers.0/Sub_1", "type": "/layers.0/Sub", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 910}, {"name": "/layers.0/Exp_1", "type": "/layers.0/Exp", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3280}, {"name": "/layers.0/ReduceSum", "type": "/layers.0/ReduceSum", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9700}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 509}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 95965}, {"name": "/layers.0/MatMul_2", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 40017}, {"name": "/layers.0/MatMul_3", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8930}, {"name": "/layers.0/Add_5", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 890}, {"name": "/layers.0/Div_1", "type": "/layers.0/Div", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8841}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1007}, {"name": "/layers.0/Mul_8", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 893}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 158075}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_6", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 85708}, {"name": "rms_norm_/layers.0/Mul_10_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5215}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 796342}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 787899}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_13", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12502}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 796088}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_9", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6893}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 21746}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_ring3_sliding_fp16.json b/benchmarks/edge_harness/results/profile_ring3_sliding_fp16.json new file mode 100644 index 000000000..d2fe0497f --- /dev/null +++ b/benchmarks/edge_harness/results/profile_ring3_sliding_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1486, "estimated_inference_peak_memory": 139599872, "first_load_time": 115214, "first_load_peak_memory": 109510656, "warm_load_time": 114073, "warm_load_peak_memory": 125624320, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [14961536, 24144032], "first_load_memory_peak_range": [19734528, 28917024], "warm_load_memory_increase_range": [0, 827344], "warm_load_memory_peak_range": [737280, 10669568], "inference_memory_increase_range": [0, 2129968], "inference_memory_peak_range": [0, 9903648], "all_compile_times": [], "all_first_load_times": [115214], "all_warm_load_times": [114073], "all_inference_times": [2310, 1720, 1648, 1682, 1698, 1625, 1628, 1619, 1561, 1514, 1490, 1986, 1815, 1857, 1898, 1837, 1814, 1797, 1788, 1727, 1612, 1642, 1615, 1601, 1504, 1627, 1506, 1494, 1531, 1499, 1491, 1513, 1499, 1565, 1510, 1493, 1497, 1500, 1494, 1496, 1499, 1488, 1494, 1505, 1488, 1494, 1493, 1488, 1493, 1493, 1491, 1520, 1523, 3012, 1500, 1512, 1507, 1502, 1495, 1501, 1496, 1493, 1508, 1494, 1533, 1518, 1497, 1508, 1520, 1501, 1503, 1500, 1499, 1501, 1497, 1577, 1749, 1645, 1707, 1657, 1586, 1646, 1564, 1676, 1557, 1587, 1524, 1519, 1594, 1578, 1517, 1517, 1586, 1545, 1501, 1543, 1492, 1516, 1486, 1496]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7431}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 121888}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18417}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_497", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 262714}, {"name": "node_Slice_500", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 69757}, {"name": "node_Slice_503", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 54734}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 488}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 242}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18138}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9962}, {"name": "node_Slice_393", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_396", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 24465}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1032}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1068}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2060}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 958}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13664}, {"name": "node_Slice_442", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_445", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1716}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7234}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 326}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2602}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 210}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 87204}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 69504}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 25594}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 415559}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 70688}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1383}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 69045}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14385}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 57686}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 475}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2854}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 190749}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 86211}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3788}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 836520}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 821543}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10772}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 821530}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6788}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17655}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_ring3_sliding_w4a16.json b/benchmarks/edge_harness/results/profile_ring3_sliding_w4a16.json new file mode 100644 index 000000000..1495f5665 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_ring3_sliding_w4a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 838, "estimated_inference_peak_memory": 138752000, "first_load_time": 104984, "first_load_peak_memory": 113438720, "warm_load_time": 104086, "warm_load_peak_memory": 125595648, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 8178736], "first_load_memory_peak_range": [7778304, 16937392], "warm_load_memory_increase_range": [0, 1753392], "warm_load_memory_peak_range": [872448, 11059632], "inference_memory_increase_range": [0, 1069104], "inference_memory_peak_range": [0, 9035184], "all_compile_times": [], "all_first_load_times": [104984], "all_warm_load_times": [104086], "all_inference_times": [1563, 1071, 986, 987, 979, 919, 891, 900, 880, 880, 874, 1063, 935, 943, 935, 941, 937, 952, 846, 867, 840, 850, 887, 883, 904, 901, 867, 868, 897, 899, 867, 867, 867, 854, 890, 953, 928, 956, 1122, 997, 1083, 1165, 908, 911, 917, 1029, 1172, 913, 953, 929, 933, 943, 960, 947, 876, 894, 953, 979, 917, 932, 917, 916, 942, 932, 962, 922, 924, 907, 862, 917, 928, 930, 933, 940, 942, 907, 930, 936, 916, 838, 845, 866, 885, 871, 873, 873, 887, 877, 966, 890, 869, 874, 862, 871, 856, 842, 872, 873, 867, 877]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 478820}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 53145}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 126}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_497", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 83913}, {"name": "node_Slice_500", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9103}, {"name": "node_Slice_503", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16251}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5974}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2653}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1451}, {"name": "node_Slice_393", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 328}, {"name": "node_Slice_396", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 355}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10257}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 492}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 865}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12237}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1279}, {"name": "node_Slice_442", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5166}, {"name": "node_Slice_445", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5196}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11283}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1134}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1710}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11689}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2282}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 198}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20515}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17531}, {"name": "/layers.0/Concat_3_output_0_converted_unsigned_symmetric", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 128046}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Transpose_output_0_converted_unsigned_symmetric", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 367695}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1231}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14654}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11444}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 57445}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 24919}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 52505}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10812}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9451}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 436131}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 403371}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 23019}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 405361}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5781}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4086}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_ring3_w4a8.json b/benchmarks/edge_harness/results/profile_ring3_w4a8.json new file mode 100644 index 000000000..c9a0a85b9 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_ring3_w4a8.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 751, "estimated_inference_peak_memory": 138072064, "first_load_time": 114182, "first_load_peak_memory": 113254400, "warm_load_time": 112303, "warm_load_peak_memory": 125153280, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2425792], "first_load_memory_peak_range": [1277952, 10691280], "warm_load_memory_increase_range": [0, 3745856], "warm_load_memory_peak_range": [3039232, 12596560], "inference_memory_increase_range": [0, 553008], "inference_memory_peak_range": [0, 9093312], "all_compile_times": [], "all_first_load_times": [114182], "all_warm_load_times": [112303], "all_inference_times": [1352, 814, 790, 796, 789, 800, 795, 858, 806, 771, 780, 919, 781, 782, 779, 833, 846, 867, 796, 777, 793, 790, 814, 793, 771, 778, 755, 756, 752, 801, 760, 761, 751, 753, 752, 752, 758, 768, 755, 762, 763, 778, 807, 822, 786, 801, 827, 763, 752, 768, 762, 784, 765, 769, 765, 789, 817, 798, 802, 769, 804, 779, 844, 806, 790, 798, 824, 776, 767, 763, 766, 764, 774, 778, 762, 781, 766, 792, 822, 1065, 1101, 1252, 967, 1195, 1185, 831, 805, 781, 824, 810, 815, 859, 826, 837, 910, 787, 780, 790, 841, 786]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 196526}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35201}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_497", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 50146}, {"name": "node_Slice_500", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14700}, {"name": "node_Slice_503", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 41331}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 435}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3404}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 288}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2107}, {"name": "node_Slice_393", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7760}, {"name": "node_Slice_396", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7794}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 614}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 322}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2028}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2098}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4610}, {"name": "node_Slice_442", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_445", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4698}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5761}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4413}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1462}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 402}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 25883}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7811}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 94386}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20565}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 21107}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11294}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 78514}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 23697}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 81863}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2495}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9500}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 464566}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10105}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 459060}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 34524}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 472323}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1068}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7103}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_ring_full_ctx1024_fp16.json b/benchmarks/edge_harness/results/profile_ring_full_ctx1024_fp16.json new file mode 100644 index 000000000..6d3e9e305 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_ring_full_ctx1024_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1567, "estimated_inference_peak_memory": 143167488, "first_load_time": 121550, "first_load_peak_memory": 114778112, "warm_load_time": 117498, "warm_load_peak_memory": 126844928, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2441456], "first_load_memory_peak_range": [1359872, 11835536], "warm_load_memory_increase_range": [0, 1691984], "warm_load_memory_peak_range": [0, 10525536], "inference_memory_increase_range": [0, 28720], "inference_memory_peak_range": [0, 9662544], "all_compile_times": [], "all_first_load_times": [121550], "all_warm_load_times": [117498], "all_inference_times": [2677, 2072, 1930, 2042, 1982, 1969, 1903, 1912, 1846, 1784, 1692, 1993, 1785, 1812, 1787, 1756, 1743, 1670, 1613, 1606, 1599, 1621, 1618, 1601, 1596, 2285, 1599, 1607, 1588, 1617, 1597, 1638, 1702, 1693, 1645, 1595, 1616, 1652, 1612, 1591, 1642, 1588, 1581, 1573, 1584, 1595, 1581, 1577, 1570, 1580, 1567, 1638, 1591, 1584, 1598, 1617, 1690, 1738, 1663, 1617, 1629, 1641, 1643, 1917, 2078, 1761, 2192, 1656, 1631, 1637, 1779, 1783, 1766, 1842, 1810, 1917, 1760, 1681, 1768, 1971, 1943, 1774, 1854, 1835, 1790, 1685, 1717, 1633, 1619, 1603, 1642, 1640, 1619, 1674, 1624, 1602, 1630, 1720, 1611, 1605]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 83386}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 171352}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 28930}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_141", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 284809}, {"name": "node_Slice_144", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 82995}, {"name": "node_Slice_147", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 48282}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 558}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 288}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 230}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7087}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7297}, {"name": "node_Slice_120", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_123", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 572}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22821}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 342}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 520}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6280}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 312}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3224}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2145}, {"name": "node_Slice_127", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_130", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_9", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_10", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3335}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10800}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 157}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2435}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 211}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 199466}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 590639}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20116}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8160}, {"name": "/layers.0/ReduceSum", "type": "/layers.0/ReduceSum", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6063}, {"name": "/layers.0/Mul_8", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3481}, {"name": "/layers.0/Concat_4", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9367}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 34369}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13540}, {"name": "node_Slice_134", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_137", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 153430}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 161754}, {"name": "/layers.0/Mul_9", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4297}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4647}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 446}, {"name": "/layers.0/Mul_10", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3998}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 159460}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_5", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 51739}, {"name": "rms_norm_/layers.0/Mul_12_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3858}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 886133}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 867797}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_15", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11363}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 862011}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_8", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8430}, {"name": "/layers.0/Concat_3_identity", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16134}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_ring_sliding_ctx1024_fp16.json b/benchmarks/edge_harness/results/profile_ring_sliding_ctx1024_fp16.json new file mode 100644 index 000000000..60b4a5ac4 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_ring_sliding_ctx1024_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1483, "estimated_inference_peak_memory": 140337152, "first_load_time": 114989, "first_load_peak_memory": 113680384, "warm_load_time": 118746, "warm_load_peak_memory": 121503744, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2375744], "first_load_memory_peak_range": [1220608, 10869504], "warm_load_memory_increase_range": [0, 4568256], "warm_load_memory_peak_range": [0, 9384288], "inference_memory_increase_range": [0, 2134064], "inference_memory_peak_range": [0, 8743104], "all_compile_times": [], "all_first_load_times": [114989], "all_warm_load_times": [118746], "all_inference_times": [2295, 1751, 1827, 1751, 1802, 1710, 1746, 1602, 1577, 1569, 1524, 1734, 1576, 1527, 1514, 1500, 1500, 1557, 1606, 1572, 1678, 1542, 1530, 1579, 1547, 1513, 1522, 1503, 1492, 1499, 1507, 1492, 1502, 1488, 1494, 1510, 1487, 1497, 1487, 1485, 1495, 1483, 1484, 1491, 1515, 1537, 1613, 1611, 1561, 1585, 1540, 1542, 1539, 1516, 1520, 1495, 1488, 1497, 1496, 1495, 1501, 1541, 1500, 1487, 1489, 1495, 1486, 1483, 1490, 1484, 1488, 1512, 1573, 1544, 1609, 1587, 1553, 1563, 1538, 1589, 1529, 1514, 1527, 1489, 1497, 1495, 1488, 1488, 1500, 1524, 1614, 1521, 1518, 1502, 1491, 1489, 1500, 1507, 1505, 1506]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 15910}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 88732}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19220}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_414", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 239352}, {"name": "node_Slice_417", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 65301}, {"name": "node_Slice_420", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 51953}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 492}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 240}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 238}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18424}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6051}, {"name": "node_Slice_430", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3276}, {"name": "node_Slice_433", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18262}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1016}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1028}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 728}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 996}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14892}, {"name": "node_Slice_437", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_440", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2710}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14963}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 396}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4478}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 221}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 134772}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 321150}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 30207}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16705}, {"name": "/layers.0/ReduceSum", "type": "/layers.0/ReduceSum", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14366}, {"name": "/layers.0/Mul_8", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3264}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1171}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 47245}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16765}, {"name": "node_Slice_444", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_447", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11471}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 88544}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 42125}, {"name": "/layers.0/Mul_9", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3701}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4128}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 438}, {"name": "/layers.0/Mul_10", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3271}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 175626}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_5", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 91027}, {"name": "rms_norm_/layers.0/Mul_12_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3511}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 834925}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 866910}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_15", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11603}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 810087}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_8", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6895}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17104}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_ring_sliding_tanh_fp16.json b/benchmarks/edge_harness/results/profile_ring_sliding_tanh_fp16.json new file mode 100644 index 000000000..91f2ee573 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_ring_sliding_tanh_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1535, "estimated_inference_peak_memory": 140099584, "first_load_time": 118480, "first_load_peak_memory": 113537024, "warm_load_time": 119765, "warm_load_peak_memory": 124760064, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 3750256], "first_load_memory_peak_range": [2940928, 12303072], "warm_load_memory_increase_range": [0, 1695824], "warm_load_memory_peak_range": [0, 10174496], "inference_memory_increase_range": [0, 2129968], "inference_memory_peak_range": [0, 9041792], "all_compile_times": [], "all_first_load_times": [118480], "all_warm_load_times": [119765], "all_inference_times": [2433, 1845, 1775, 1750, 1746, 1712, 1710, 1617, 1608, 1599, 1592, 2135, 1871, 1768, 1643, 1652, 1625, 1601, 1642, 1628, 1594, 1599, 1586, 1684, 1596, 1673, 1794, 1720, 1643, 1633, 1785, 1822, 1792, 1709, 1677, 1608, 1569, 1561, 1567, 1559, 1559, 1588, 1585, 1571, 1599, 1555, 1553, 1599, 1702, 1771, 1831, 1741, 1813, 1716, 1891, 1808, 1710, 1573, 1564, 1568, 1555, 1546, 1557, 1547, 1544, 1550, 1535, 1550, 1578, 1621, 1620, 1611, 1576, 1583, 1640, 1649, 1675, 1589, 1605, 1734, 1707, 1830, 1744, 1761, 1783, 1618, 1707, 1621, 1652, 1712, 1616, 1601, 1605, 1669, 1593, 1659, 1790, 1640, 1628, 1598]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16153}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 110098}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19199}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_180", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 250822}, {"name": "node_Slice_183", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 73962}, {"name": "node_Slice_186", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 60326}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 468}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 232}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18026}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4499}, {"name": "node_Slice_331", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2506}, {"name": "node_Slice_334", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19291}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1037}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2438}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2102}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 975}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14896}, {"name": "node_Slice_338", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_341", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2937}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11155}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 345}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5722}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 267}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 130807}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 335671}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 36543}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13766}, {"name": "/layers.0/ReduceSum", "type": "/layers.0/ReduceSum", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14631}, {"name": "/layers.0/Mul_8", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5612}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1141}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 47450}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10622}, {"name": "node_Slice_345", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_348", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 87913}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 40921}, {"name": "/layers.0/MatMul_2", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 37973}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4267}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 416}, {"name": "/layers.0/Mul_9", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3650}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 176300}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_5", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 86132}, {"name": "rms_norm_/layers.0/Mul_11_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4391}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 823660}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 824889}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_18", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12179}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 834809}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_9", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6887}, {"name": "/layers.0/Add_2_identity", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22903}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_full_1024_fp16.json b/benchmarks/edge_harness/results/profile_spark_full_1024_fp16.json new file mode 100644 index 000000000..dd394e204 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_full_1024_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1559, "estimated_inference_peak_memory": 142434304, "first_load_time": 111668, "first_load_peak_memory": 105967616, "warm_load_time": 114274, "warm_load_peak_memory": 125005824, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 7911872], "first_load_memory_peak_range": [0, 8989584], "warm_load_memory_increase_range": [0, 2039504], "warm_load_memory_peak_range": [0, 10238496], "inference_memory_increase_range": [0, 36912], "inference_memory_peak_range": [0, 8819840], "all_compile_times": [], "all_first_load_times": [111668], "all_warm_load_times": [114274], "all_inference_times": [2843, 2228, 2052, 1796, 1909, 1731, 1743, 1744, 1772, 1735, 1709, 2235, 2071, 2200, 1893, 1750, 1730, 1763, 1742, 1731, 1727, 1699, 1902, 2682, 1716, 1695, 1750, 1714, 1825, 1681, 1637, 1656, 1635, 1627, 1641, 1620, 1676, 1629, 1640, 1611, 1634, 1657, 1593, 1666, 1585, 1573, 1592, 1578, 1568, 1568, 1572, 1572, 1565, 1575, 1634, 1573, 1568, 1565, 1582, 1562, 1577, 1575, 1593, 1578, 1564, 1576, 1574, 1618, 1630, 2442, 1601, 1653, 1571, 1594, 1579, 1578, 1572, 1582, 1562, 1565, 1561, 1593, 1565, 1563, 1583, 1570, 1617, 1562, 1559, 1572, 1660, 1762, 1855, 1703, 1585, 1563, 1563, 1574, 1569, 1559]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 69254}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 97184}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20589}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_197", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 243001}, {"name": "node_Slice_200", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 94930}, {"name": "node_Slice_203", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 55020}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1211}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 246}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1777}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2550}, {"name": "node_Slice_398", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_401", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 554}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18138}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 318}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3683}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4358}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 299}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3673}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2072}, {"name": "node_Slice_380", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_383", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_9", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_10", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2262}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12207}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 124}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3133}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 795}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 165841}, {"name": "/layers.0/Concat_4", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 39039}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 143557}, {"name": "/layers.0/Concat_5", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 36910}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 193}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 511661}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 78410}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5514}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 82942}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 24756}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 169726}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1013}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1759}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 159736}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 65762}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3311}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 816353}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 811019}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13251}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 814625}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7117}, {"name": "/layers.0/Concat_3_identity", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16593}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_full_1024_w4a16.json b/benchmarks/edge_harness/results/profile_spark_full_1024_w4a16.json new file mode 100644 index 000000000..e5766aaa8 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_full_1024_w4a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1127, "estimated_inference_peak_memory": 140668928, "first_load_time": 104370, "first_load_peak_memory": 110632960, "warm_load_time": 109320, "warm_load_peak_memory": 126144512, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 4232976], "first_load_memory_peak_range": [0, 9456560], "warm_load_memory_increase_range": [8745632, 18233216], "warm_load_memory_peak_range": [17854464, 27342048], "inference_memory_increase_range": [0, 2125872], "inference_memory_peak_range": [0, 9296496], "all_compile_times": [], "all_first_load_times": [104370], "all_warm_load_times": [109320], "all_inference_times": [1718, 1166, 1139, 1136, 1138, 1144, 1139, 1134, 1133, 1131, 1139, 1286, 1170, 1169, 1149, 1174, 1159, 1187, 1202, 1187, 1181, 2016, 1148, 1158, 1153, 1147, 1143, 1149, 1137, 1142, 1136, 1149, 1140, 1134, 1181, 1150, 1152, 1147, 1159, 1146, 1325, 1507, 1294, 1464, 1160, 1149, 1153, 1157, 1167, 1159, 1165, 1163, 1188, 1206, 1221, 1214, 1219, 1205, 1189, 1164, 1167, 1162, 1172, 1148, 1146, 1153, 1140, 1142, 1192, 1138, 1137, 1131, 1148, 1140, 1140, 1131, 1129, 1140, 1129, 1129, 1133, 1137, 1130, 1130, 1135, 1130, 1133, 1130, 1134, 1134, 1130, 1174, 1135, 1137, 1130, 1140, 1129, 1127, 1127, 1134]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 748976}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 31643}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 859}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_197", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 120763}, {"name": "node_Slice_200", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8684}, {"name": "node_Slice_203", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12268}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4244}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8100}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5752}, {"name": "node_Slice_398", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 349}, {"name": "node_Slice_401", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 471}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 383}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 444}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4160}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 238}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 918}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3921}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 764}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 795}, {"name": "node_Slice_380", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4592}, {"name": "node_Slice_383", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4539}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_9", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4613}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_10", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4925}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19821}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1243}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2842}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2596}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4009}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 851}, {"name": "/layers.0/Concat_4", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 29145}, {"name": "/layers.0/Concat_5", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Concat_5_output_0_converted_unsigned_symmetric", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 392667}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 278}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Transpose_output_0_converted_unsigned_symmetric", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 753404}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2731}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 76697}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17172}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 599083}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 45795}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 30840}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18077}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13617}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 502662}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 456678}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 56976}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 441442}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5791}, {"name": "/layers.0/Concat_3_identity", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11912}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_full_1024_w8a16.json b/benchmarks/edge_harness/results/profile_spark_full_1024_w8a16.json new file mode 100644 index 000000000..ecafaf5b9 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_full_1024_w8a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1129, "estimated_inference_peak_memory": 139857920, "first_load_time": 108126, "first_load_peak_memory": 111996928, "warm_load_time": 103320, "warm_load_peak_memory": 124866560, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2414112], "first_load_memory_peak_range": [0, 9691168], "warm_load_memory_increase_range": [0, 2343088], "warm_load_memory_peak_range": [405504, 10119408], "inference_memory_increase_range": [0, 2125872], "inference_memory_peak_range": [0, 8857072], "all_compile_times": [], "all_first_load_times": [108126], "all_warm_load_times": [103320], "all_inference_times": [1877, 1227, 1193, 1143, 1150, 1137, 1140, 1135, 1137, 1140, 1136, 1400, 1252, 1232, 1255, 1247, 1269, 1245, 1252, 1238, 1241, 1192, 1175, 1167, 1172, 1166, 1173, 1152, 1203, 1139, 1137, 1184, 1145, 1136, 1140, 1141, 1135, 1131, 1131, 1142, 1130, 1138, 1134, 1134, 1133, 1135, 1133, 1137, 1134, 1140, 1131, 1134, 1136, 1135, 1144, 1141, 1130, 1129, 1129, 1137, 1132, 1136, 1130, 1131, 1136, 1151, 1138, 1139, 1134, 1142, 1143, 1460, 1141, 1160, 1147, 1137, 1134, 1155, 1134, 1133, 1140, 1134, 1132, 1137, 1133, 1132, 1132, 1135, 1134, 1163, 1253, 1203, 1226, 1264, 1187, 1185, 1167, 1201, 1175, 1165]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 753219}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 28894}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 755}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_197", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 105271}, {"name": "node_Slice_200", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6085}, {"name": "node_Slice_203", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17677}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5538}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2106}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3072}, {"name": "node_Slice_398", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 528}, {"name": "node_Slice_401", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 654}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 607}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 584}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3506}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 228}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 899}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3427}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 549}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1021}, {"name": "node_Slice_380", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1066}, {"name": "node_Slice_383", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1029}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_9", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1071}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_10", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2146}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6818}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 939}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1964}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3212}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1028}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 919}, {"name": "/layers.0/Concat_4", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17605}, {"name": "/layers.0/Concat_5", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Concat_5_output_0_converted_unsigned_symmetric", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 462750}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 290}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Transpose_output_0_converted_unsigned_symmetric", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 734291}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2296}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 34398}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13723}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 599222}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19554}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22649}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20985}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16477}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 433795}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 420307}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22637}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 426424}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5772}, {"name": "/layers.0/Concat_3_identity", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5113}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_full_4096_fp16.json b/benchmarks/edge_harness/results/profile_spark_full_4096_fp16.json new file mode 100644 index 000000000..c6a0bf48e --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_full_4096_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 2964, "estimated_inference_peak_memory": 165826560, "first_load_time": 117398, "first_load_peak_memory": 116396032, "warm_load_time": 119192, "warm_load_peak_memory": 128565248, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2872272], "first_load_memory_peak_range": [1818624, 11197040], "warm_load_memory_increase_range": [0, 1570160], "warm_load_memory_peak_range": [946176, 10568032], "inference_memory_increase_range": [0, 161936], "inference_memory_peak_range": [8417280, 17976128], "all_compile_times": [], "all_first_load_times": [117398], "all_warm_load_times": [119192], "all_inference_times": [4189, 4096, 3274, 3330, 3502, 3357, 3291, 3243, 3201, 3240, 3344, 3745, 3529, 3143, 3181, 3135, 3129, 3103, 3085, 3095, 3116, 3122, 3066, 3086, 3055, 3048, 3141, 3331, 3308, 3225, 3189, 3152, 3101, 3110, 3204, 3093, 3197, 3042, 3050, 3037, 3020, 3033, 3035, 3048, 3024, 3028, 3013, 3010, 3017, 3017, 3081, 3116, 3029, 3016, 3017, 3003, 3009, 3006, 3011, 3013, 3035, 2997, 3024, 3011, 3000, 2992, 3040, 3003, 2995, 3038, 3018, 2988, 2992, 2978, 3018, 2992, 2984, 2984, 3094, 3060, 2992, 3078, 3099, 3002, 2997, 3012, 2976, 2977, 2983, 2978, 2974, 2984, 2975, 2973, 3004, 2977, 2975, 2967, 2994, 2964]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1269995}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 134163}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16521}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_197", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 611448}, {"name": "node_Slice_200", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 177324}, {"name": "node_Slice_203", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 170897}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 552}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 446}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 684}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9884}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4009}, {"name": "node_Slice_398", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_401", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1403}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9730}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 327}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 579}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9690}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 338}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9710}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1441}, {"name": "node_Slice_380", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_383", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_9", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_10", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3211}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18997}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 159}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1817}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 561388}, {"name": "/layers.0/Concat_4", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1305384}, {"name": "/layers.0/Concat_5", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 400832}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 233}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1911875}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 104156}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_full_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 24472}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 138055}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 32128}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4851959}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9552}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5168}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 187870}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 134932}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 26788}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 749979}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 755165}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11232}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 763836}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6915}, {"name": "/layers.0/Concat_3_identity", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17325}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_full_4096_w4a16.json b/benchmarks/edge_harness/results/profile_spark_full_4096_w4a16.json new file mode 100644 index 000000000..9e9ea2199 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_full_4096_w4a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1662, "estimated_inference_peak_memory": 150540288, "first_load_time": 105164, "first_load_peak_memory": 112205824, "warm_load_time": 104154, "warm_load_peak_memory": 125681664, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2648752], "first_load_memory_peak_range": [0, 9906208], "warm_load_memory_increase_range": [0, 1723632], "warm_load_memory_peak_range": [909312, 10868704], "inference_memory_increase_range": [0, 0], "inference_memory_peak_range": [4214784, 13014736], "all_compile_times": [], "all_first_load_times": [105164], "all_warm_load_times": [104154], "all_inference_times": [2520, 1893, 1846, 1825, 2000, 2258, 2145, 2024, 1988, 1839, 1920, 1953, 1829, 1831, 1846, 1850, 1843, 1868, 1807, 1778, 1768, 1789, 1801, 1749, 1760, 1759, 1740, 1752, 1781, 1754, 1730, 1719, 1734, 1723, 1725, 1804, 1743, 1732, 1771, 1750, 1714, 1753, 1754, 1700, 1730, 1757, 1711, 1824, 2229, 1700, 1701, 1751, 1752, 1704, 1741, 1726, 1701, 1698, 1724, 1715, 1858, 1931, 1833, 1783, 1766, 1788, 1744, 1745, 1713, 1719, 1777, 1840, 2597, 1821, 1686, 1729, 1757, 1707, 1669, 1739, 1742, 1680, 1702, 1741, 1731, 1678, 1668, 1702, 1665, 1710, 1668, 1717, 1670, 1662, 1686, 1733, 1680, 1782, 1699, 1766]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2600475}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 45030}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1515}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_197", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 119563}, {"name": "node_Slice_200", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 25753}, {"name": "node_Slice_203", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14909}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8447}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2725}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2667}, {"name": "node_Slice_398", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 332}, {"name": "node_Slice_401", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 593}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 337}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 417}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3293}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 156}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 981}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4865}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 683}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 804}, {"name": "node_Slice_380", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 292}, {"name": "node_Slice_383", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1359}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_9", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 318}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_10", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2273}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18514}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2160}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1459}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2512}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2744}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 855}, {"name": "/layers.0/Concat_4", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Concat_5", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Concat_5_output_0_converted_unsigned_symmetric", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1383760}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 326}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Transpose_output_0_converted_unsigned_symmetric", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2445931}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10835}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 65259}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 29429}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2483345}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 647}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 48454}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 26674}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 21363}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11673}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 426236}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 396398}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 56947}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 416298}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5781}, {"name": "/layers.0/Concat_3_identity", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6195}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_full_4096_w8a16.json b/benchmarks/edge_harness/results/profile_spark_full_4096_w8a16.json new file mode 100644 index 000000000..650528be8 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_full_4096_w8a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1681, "estimated_inference_peak_memory": 150032384, "first_load_time": 105307, "first_load_peak_memory": 113418240, "warm_load_time": 113514, "warm_load_peak_memory": 125550592, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [0, 2382448], "first_load_memory_peak_range": [1306624, 10690688], "warm_load_memory_increase_range": [45906848, 55550544], "warm_load_memory_peak_range": [55463936, 65107632], "inference_memory_increase_range": [0, 0], "inference_memory_peak_range": [4222976, 13536400], "all_compile_times": [], "all_first_load_times": [105307], "all_warm_load_times": [113514], "all_inference_times": [2541, 1910, 1902, 1847, 1815, 1825, 1816, 1791, 1804, 1798, 1806, 2038, 1960, 1885, 1869, 1845, 1840, 1803, 1761, 1762, 1755, 1749, 1738, 1759, 1738, 1730, 1741, 1729, 1732, 1727, 1730, 1716, 1720, 1721, 1716, 1725, 1705, 1712, 1720, 1698, 1721, 1701, 1695, 1702, 1704, 1699, 1708, 1702, 1697, 1701, 1691, 1693, 1697, 1691, 1691, 1692, 1703, 1688, 1690, 1697, 1688, 1687, 1685, 1687, 1685, 1684, 1706, 1692, 1690, 1696, 1695, 1685, 1693, 1690, 1685, 1690, 1685, 1694, 1689, 1681, 1694, 1756, 1703, 1692, 1687, 1704, 1693, 1720, 1719, 1724, 1768, 2409, 2041, 2156, 2021, 1791, 1771, 1774, 1756, 1759]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2602337}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 54251}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1015}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_197", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 112641}, {"name": "node_Slice_200", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19808}, {"name": "node_Slice_203", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 14232}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9755}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1859}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3080}, {"name": "node_Slice_398", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 351}, {"name": "node_Slice_401", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 474}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_5", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 325}, {"name": "fused_/layers.0/Slice_3_/layers.0/Slice_6", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 507}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3629}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 180}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 824}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3434}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2140}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1004}, {"name": "node_Slice_380", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 403}, {"name": "node_Slice_383", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1237}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_9", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1004}, {"name": "fused_/layers.0/Slice_7_/layers.0/Slice_10", "type": "fused", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1032}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16423}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1645}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2196}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3947}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2951}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 868}, {"name": "/layers.0/Concat_4", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Concat_5", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Concat_5_output_0_converted_unsigned_symmetric", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1369699}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 330}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Transpose_output_0_converted_unsigned_symmetric", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2507583}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13418}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 61244}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 30559}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2486361}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 641}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 64936}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 30047}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 24139}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11677}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 448594}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 394511}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 65337}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 404610}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5793}, {"name": "/layers.0/Concat_3_identity", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2_identity", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7435}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_sliding_1024_fp16.json b/benchmarks/edge_harness/results/profile_spark_sliding_1024_fp16.json new file mode 100644 index 000000000..54c95021f --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_sliding_1024_fp16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 1689, "estimated_inference_peak_memory": 142680064, "first_load_time": 121570, "first_load_peak_memory": 113696768, "warm_load_time": 137569, "warm_load_peak_memory": 128036864, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [10469488, 19386816], "first_load_memory_peak_range": [16543744, 25461072], "warm_load_memory_increase_range": [0, 1551408], "warm_load_memory_peak_range": [1003520, 10963472], "inference_memory_increase_range": [0, 2142256], "inference_memory_peak_range": [0, 9338912], "all_compile_times": [], "all_first_load_times": [121570], "all_warm_load_times": [137569], "all_inference_times": [2808, 2255, 1815, 1832, 1819, 1771, 1750, 1735, 1742, 1713, 1834, 1799, 1878, 1817, 1775, 1755, 1761, 1710, 1709, 1738, 1710, 1742, 1718, 1708, 1801, 1734, 1696, 1703, 1716, 1707, 1715, 1700, 1720, 1705, 1723, 1711, 1694, 1717, 1705, 1730, 1709, 1701, 1714, 1718, 1719, 1701, 1695, 1714, 1692, 1706, 1793, 1810, 1709, 1709, 1700, 1707, 1695, 1833, 1743, 1696, 1714, 1695, 1710, 1701, 1689, 1710, 1702, 1702, 1698, 1700, 1700, 1690, 1707, 1726, 1695, 1696, 1720, 1705, 1711, 1710, 1691, 1734, 1703, 1700, 1854, 1857, 1883, 1732, 1820, 1788, 1847, 1768, 1756, 1793, 1751, 1766, 1736, 1785, 1823, 1709]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22752}, {"name": "QNN_DATATYPE_FLOAT_32_converted_x_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 177194}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11568}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_167", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 262401}, {"name": "node_Slice_170", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 51437}, {"name": "node_Slice_173", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 57400}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 450}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 257}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 8613}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9012}, {"name": "node_Slice_307", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_310", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_cos_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 32027}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1014}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1020}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_sin_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 292}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 970}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17279}, {"name": "node_Slice_314", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_317", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4937}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 16538}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 329}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7043}, {"name": "QNN_DATATYPE_FLOAT_32_converted_k_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 72387}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_32_converted_v_cache_0_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 75325}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Slice_7", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 35932}, {"name": "/layers.0/Slice_8", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 83475}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 203}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 352855}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 58887}, {"name": "QNN_DATATYPE_FLOAT_32_converted_mask_sw_QNN_DATATYPE_FLOAT_16", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1234}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 67776}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19942}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 56116}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 408}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2904}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 207054}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 71853}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3914}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 903120}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 890467}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11613}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 855256}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7318}, {"name": "/layers.0/Slice_7_identity", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Slice_8_identity", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_0_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_1_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 112587}, {"name": "QNN_DATATYPE_FLOAT_16_converted_output_2_QNN_DATATYPE_FLOAT_32", "type": "QNN", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 191600}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 107358}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_sliding_1024_w4a16.json b/benchmarks/edge_harness/results/profile_spark_sliding_1024_w4a16.json new file mode 100644 index 000000000..f6aefd6b8 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_sliding_1024_w4a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 917, "estimated_inference_peak_memory": 140255232, "first_load_time": 106880, "first_load_peak_memory": 114032640, "warm_load_time": 110397, "warm_load_peak_memory": 126791680, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [7005456, 16175344], "first_load_memory_peak_range": [15106048, 24275936], "warm_load_memory_increase_range": [0, 1740688], "warm_load_memory_peak_range": [872448, 11080000], "inference_memory_increase_range": [0, 1077296], "inference_memory_peak_range": [0, 9065936], "all_compile_times": [], "all_first_load_times": [106880], "all_warm_load_times": [110397], "all_inference_times": [1700, 1223, 1139, 1136, 1147, 1075, 1086, 1058, 1032, 1065, 1067, 1187, 1027, 1001, 1011, 981, 1012, 990, 928, 927, 943, 936, 932, 937, 929, 923, 940, 932, 925, 923, 977, 941, 932, 922, 923, 925, 923, 922, 922, 948, 926, 935, 945, 967, 1355, 1141, 963, 939, 937, 952, 931, 960, 959, 937, 940, 946, 930, 931, 929, 925, 922, 971, 925, 934, 923, 919, 923, 917, 923, 918, 924, 921, 932, 932, 963, 922, 922, 934, 927, 921, 922, 932, 929, 930, 934, 940, 938, 933, 950, 989, 969, 1030, 925, 936, 1013, 961, 954, 998, 1001, 971]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 434706}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18850}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 560}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_167", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 110003}, {"name": "node_Slice_170", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6794}, {"name": "node_Slice_173", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 27879}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4968}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1714}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1166}, {"name": "node_Slice_307", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 445}, {"name": "node_Slice_310", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 549}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10392}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 525}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 960}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11076}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1155}, {"name": "node_Slice_314", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 361}, {"name": "node_Slice_317", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 411}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 15403}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1153}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1514}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 7266}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6844}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Slice_7", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Slice_8", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 102437}, {"name": "output_2_identity_converted_unsigned_symmetric", "type": "output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 100768}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 241}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Transpose_output_0_converted_unsigned_symmetric", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 589399}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 548}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19325}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13870}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 56867}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 23046}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 20241}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 15582}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 13552}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 436960}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 413544}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 33240}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 420145}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5788}, {"name": "/layers.0/Slice_7_identity", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Slice_8_identity", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 323697}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_spark_sliding_1024_w8a16.json b/benchmarks/edge_harness/results/profile_spark_sliding_1024_w8a16.json new file mode 100644 index 000000000..67a26e9ea --- /dev/null +++ b/benchmarks/edge_harness/results/profile_spark_sliding_1024_w8a16.json @@ -0,0 +1 @@ +{"execution_summary": {"estimated_inference_time": 912, "estimated_inference_peak_memory": 139661312, "first_load_time": 109739, "first_load_peak_memory": 111988736, "warm_load_time": 112693, "warm_load_peak_memory": 126304256, "compile_time": 0, "compile_peak_memory": 0, "compile_memory_increase_range": [0, 0], "compile_memory_peak_range": [0, 0], "first_load_memory_increase_range": [956848, 10385104], "first_load_memory_peak_range": [7315456, 16743712], "warm_load_memory_increase_range": [0, 1430080], "warm_load_memory_peak_range": [1204224, 11008608], "inference_memory_increase_range": [0, 1081392], "inference_memory_peak_range": [0, 9321968], "all_compile_times": [], "all_first_load_times": [109739], "all_warm_load_times": [112693], "all_inference_times": [1573, 1015, 968, 979, 1034, 984, 969, 975, 995, 1011, 1039, 1116, 964, 952, 948, 973, 943, 943, 945, 964, 945, 938, 940, 950, 920, 919, 916, 918, 929, 919, 915, 919, 941, 928, 923, 916, 923, 925, 920, 920, 915, 918, 921, 915, 912, 916, 917, 921, 926, 922, 947, 945, 931, 937, 946, 934, 934, 925, 932, 928, 1166, 1281, 1060, 1280, 1007, 965, 945, 937, 949, 947, 999, 947, 957, 955, 947, 1003, 953, 958, 985, 954, 956, 954, 953, 949, 943, 955, 945, 935, 936, 935, 972, 919, 915, 921, 914, 921, 915, 1004, 920, 930]}, "execution_detail": [{"name": "Input", "type": "Input", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 424987}, {"name": "rms_norm_/layers.0/Mul_1_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 22590}, {"name": "OpId_20", "type": "", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 357}, {"name": "/layers.0/q_k_v_proj/MatMul", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/q_k_v_proj/MatMul_post_reshape", "type": "/layers.0/q", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Slice_167", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 110316}, {"name": "node_Slice_170", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 17183}, {"name": "node_Slice_173", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 18351}, {"name": "/layers.0/Reshape", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_1", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Reshape_2", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/g_proj/MatMul", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 4766}, {"name": "/layers.0/g_proj/MatMul_post_reshape", "type": "/layers.0/g", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Sigmoid", "type": "/layers.0/Sigmoid", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1702}, {"name": "/layers.0/Reshape_3", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1181}, {"name": "node_Slice_307", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 519}, {"name": "node_Slice_310", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 509}, {"name": "/layers.0/Mul_2", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9841}, {"name": "/layers.0/Neg", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 935}, {"name": "/layers.0/Concat", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 980}, {"name": "/layers.0/Mul_3", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 10511}, {"name": "/layers.0/Add_1", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1276}, {"name": "node_Slice_314", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 3532}, {"name": "node_Slice_317", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2044}, {"name": "/layers.0/Mul_4", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11814}, {"name": "/layers.0/Neg_1", "type": "/layers.0/Neg", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1025}, {"name": "/layers.0/Concat_1", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2834}, {"name": "/layers.0/Mul_5", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 6396}, {"name": "/layers.0/Add_2", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 2942}, {"name": "/layers.0/Concat_2", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Concat_3", "type": "/layers.0/Concat", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Slice_7", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Slice_8", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 98248}, {"name": "output_2_identity_converted_unsigned_symmetric", "type": "output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 100678}, {"name": "/layers.0/Reshape_4", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 225}, {"name": "/layers.0/Transpose", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Transpose_output_0_converted_unsigned_symmetric", "type": "/layers.0/Transpose", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/MatMul", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 580442}, {"name": "/layers.0/Mul_6", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 1548}, {"name": "/layers.0/Add_3", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19474}, {"name": "/layers.0/Softmax", "type": "/layers.0/Softmax", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 11352}, {"name": "/layers.0/MatMul_1", "type": "/layers.0/MatMul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 54900}, {"name": "/layers.0/Reshape_5", "type": "/layers.0/Reshape", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_7", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 24861}, {"name": "/layers.0/out_proj/MatMul", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 19850}, {"name": "/layers.0/out_proj/MatMul_post_reshape", "type": "/layers.0/out", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_4", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 12158}, {"name": "rms_norm_/layers.0/Mul_9_output_0", "type": "rms", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 9906}, {"name": "/layers.0/gate_proj/MatMul", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/gate_proj/MatMul_post_reshape", "type": "/layers.0/gate", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "node_Gelu_0", "type": "node", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 449665}, {"name": "/layers.0/up_proj/MatMul", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 444697}, {"name": "/layers.0/up_proj/MatMul_post_reshape", "type": "/layers.0/up", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Mul_12", "type": "/layers.0/Mul", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 30779}, {"name": "/layers.0/down_proj/MatMul", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 438793}, {"name": "/layers.0/down_proj/MatMul_post_reshape", "type": "/layers.0/down", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Add_7", "type": "/layers.0/Add", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 5796}, {"name": "/layers.0/Slice_7_identity", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "/layers.0/Slice_8_identity", "type": "/layers.0/Slice", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 0}, {"name": "Output", "type": "Output", "compute_unit": "NPU", "execution_time": 0, "execution_cycles": 330825}]} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_w4a16_g128_p1.json b/benchmarks/edge_harness/results/profile_w4a16_g128_p1.json new file mode 100644 index 000000000..703cd1f69 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_w4a16_g128_p1.json @@ -0,0 +1,224 @@ +{ + "tag": "bfloat16/None/eager=False", + "wall_step_ms": 12.744, + "tps": 78.47, + "profiled_total_ms_per_step": 12.345, + "buckets_ms_per_step": { + "gemm": 7.112, + "framework": 2.807, + "other_op": 1.489, + "attn": 0.937 + }, + "top": [ + { + "op": "aten::_weight_int4pack_mm_for_cpu", + "bucket": "gemm", + "ms_per_step": 6.5937, + "calls_per_step": 113.0 + }, + { + "op": "## Call CompiledFxGraph None ##", + "bucket": "framework", + "ms_per_step": 2.0585, + "calls_per_step": 1.0 + }, + { + "op": "vllm::cpu_int4_linear", + "bucket": "gemm", + "ms_per_step": 0.5181, + "calls_per_step": 113.0 + }, + { + "op": "vllm::unified_attention_with_output", + "bucket": "attn", + "ms_per_step": 0.4498, + "calls_per_step": 28.0 + }, + { + "op": "vllm::unified_kv_cache_update", + "bucket": "framework", + "ms_per_step": 0.4096, + "calls_per_step": 28.0 + }, + { + "op": "_C::cpu_attention_with_kv_cache", + "bucket": "attn", + "ms_per_step": 0.3554, + "calls_per_step": 28.0 + }, + { + "op": "aten::argmax", + "bucket": "other_op", + "ms_per_step": 0.2452, + "calls_per_step": 1.0 + }, + { + "op": "aten::copy_", + "bucket": "other_op", + "ms_per_step": 0.2452, + "calls_per_step": 75.0 + }, + { + "op": "vllm::spark_head_gate", + "bucket": "framework", + "ms_per_step": 0.2093, + "calls_per_step": 28.0 + }, + { + "op": "aten::slice", + "bucket": "other_op", + "ms_per_step": 0.1452, + "calls_per_step": 230.0 + }, + { + "op": "_C::cpu_attn_reshape_and_cache", + "bucket": "attn", + "ms_per_step": 0.1316, + "calls_per_step": 28.0 + }, + { + "op": "aten::mul", + "bucket": "other_op", + "ms_per_step": 0.1275, + "calls_per_step": 28.0 + }, + { + "op": "Pregraph bytecode", + "bucket": "framework", + "ms_per_step": 0.1237, + "calls_per_step": 1.0 + }, + { + "op": "aten::sum", + "bucket": "other_op", + "ms_per_step": 0.094, + "calls_per_step": 28.0 + }, + { + "op": "aten::as_strided", + "bucket": "other_op", + "ms_per_step": 0.0889, + "calls_per_step": 290.0 + }, + { + "op": "aten::narrow", + "bucket": "other_op", + "ms_per_step": 0.0759, + "calls_per_step": 112.0 + }, + { + "op": "aten::_to_copy", + "bucket": "other_op", + "ms_per_step": 0.0736, + "calls_per_step": 61.0 + }, + { + "op": "aten::chunk", + "bucket": "other_op", + "ms_per_step": 0.0566, + "calls_per_step": 56.0 + }, + { + "op": "aten::split", + "bucket": "other_op", + "ms_per_step": 0.0512, + "calls_per_step": 56.0 + }, + { + "op": "aten::view", + "bucket": "other_op", + "ms_per_step": 0.0471, + "calls_per_step": 62.0 + }, + { + "op": "aten::empty", + "bucket": "other_op", + "ms_per_step": 0.0455, + "calls_per_step": 145.0 + }, + { + "op": "aten::to", + "bucket": "other_op", + "ms_per_step": 0.0313, + "calls_per_step": 64.0 + }, + { + "op": "aten::empty_strided", + "bucket": "other_op", + "ms_per_step": 0.0267, + "calls_per_step": 61.0 + }, + { + "op": "aten::fill_", + "bucket": "other_op", + "ms_per_step": 0.0223, + "calls_per_step": 40.0 + }, + { + "op": "aten::index", + "bucket": "other_op", + "ms_per_step": 0.0188, + "calls_per_step": 2.0 + }, + { + "op": "aten::unsqueeze", + "bucket": "other_op", + "ms_per_step": 0.0177, + "calls_per_step": 29.0 + }, + { + "op": "_C::compute_slot_mapping_kernel_impl", + "bucket": "other_op", + "ms_per_step": 0.0167, + "calls_per_step": 4.0 + }, + { + "op": "_C::get_scheduler_metadata", + "bucket": "other_op", + "ms_per_step": 0.0149, + "calls_per_step": 4.0 + }, + { + "op": "aten::new_empty", + "bucket": "other_op", + "ms_per_step": 0.0139, + "calls_per_step": 28.0 + }, + { + "op": "aten::add", + "bucket": "other_op", + "ms_per_step": 0.0091, + "calls_per_step": 3.0 + }, + { + "op": "AOTDispatcher Runtime Wrapper Prologue", + "bucket": "framework", + "ms_per_step": 0.006, + "calls_per_step": 1.0 + }, + { + "op": "aten::sub", + "bucket": "other_op", + "ms_per_step": 0.0052, + "calls_per_step": 1.0 + }, + { + "op": "aten::lt", + "bucket": "other_op", + "ms_per_step": 0.0035, + "calls_per_step": 1.0 + }, + { + "op": "aten::index_select", + "bucket": "other_op", + "ms_per_step": 0.0031, + "calls_per_step": 1.0 + }, + { + "op": "aten::detach", + "bucket": "other_op", + "ms_per_step": 0.0025, + "calls_per_step": 2.0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_w4a16_g128_p2.json b/benchmarks/edge_harness/results/profile_w4a16_g128_p2.json new file mode 100644 index 000000000..ecd24f265 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_w4a16_g128_p2.json @@ -0,0 +1,224 @@ +{ + "tag": "bfloat16/None/eager=False", + "wall_step_ms": 12.597, + "tps": 79.39, + "profiled_total_ms_per_step": 12.108, + "buckets_ms_per_step": { + "gemm": 7.05, + "framework": 2.697, + "other_op": 1.447, + "attn": 0.914 + }, + "top": [ + { + "op": "aten::_weight_int4pack_mm_for_cpu", + "bucket": "gemm", + "ms_per_step": 6.5424, + "calls_per_step": 113.0 + }, + { + "op": "## Call CompiledFxGraph None ##", + "bucket": "framework", + "ms_per_step": 1.968, + "calls_per_step": 1.0 + }, + { + "op": "vllm::cpu_int4_linear", + "bucket": "gemm", + "ms_per_step": 0.5072, + "calls_per_step": 113.0 + }, + { + "op": "vllm::unified_attention_with_output", + "bucket": "attn", + "ms_per_step": 0.4379, + "calls_per_step": 28.0 + }, + { + "op": "vllm::unified_kv_cache_update", + "bucket": "framework", + "ms_per_step": 0.3948, + "calls_per_step": 28.0 + }, + { + "op": "_C::cpu_attention_with_kv_cache", + "bucket": "attn", + "ms_per_step": 0.3471, + "calls_per_step": 28.0 + }, + { + "op": "aten::argmax", + "bucket": "other_op", + "ms_per_step": 0.2449, + "calls_per_step": 1.0 + }, + { + "op": "aten::copy_", + "bucket": "other_op", + "ms_per_step": 0.2324, + "calls_per_step": 75.0 + }, + { + "op": "vllm::spark_head_gate", + "bucket": "framework", + "ms_per_step": 0.2022, + "calls_per_step": 28.0 + }, + { + "op": "aten::slice", + "bucket": "other_op", + "ms_per_step": 0.1406, + "calls_per_step": 230.0 + }, + { + "op": "_C::cpu_attn_reshape_and_cache", + "bucket": "attn", + "ms_per_step": 0.1287, + "calls_per_step": 28.0 + }, + { + "op": "Pregraph bytecode", + "bucket": "framework", + "ms_per_step": 0.1258, + "calls_per_step": 1.0 + }, + { + "op": "aten::mul", + "bucket": "other_op", + "ms_per_step": 0.1222, + "calls_per_step": 28.0 + }, + { + "op": "aten::sum", + "bucket": "other_op", + "ms_per_step": 0.091, + "calls_per_step": 28.0 + }, + { + "op": "aten::as_strided", + "bucket": "other_op", + "ms_per_step": 0.0874, + "calls_per_step": 290.0 + }, + { + "op": "aten::_to_copy", + "bucket": "other_op", + "ms_per_step": 0.0742, + "calls_per_step": 61.0 + }, + { + "op": "aten::narrow", + "bucket": "other_op", + "ms_per_step": 0.0694, + "calls_per_step": 112.0 + }, + { + "op": "aten::chunk", + "bucket": "other_op", + "ms_per_step": 0.056, + "calls_per_step": 56.0 + }, + { + "op": "aten::split", + "bucket": "other_op", + "ms_per_step": 0.0497, + "calls_per_step": 56.0 + }, + { + "op": "aten::view", + "bucket": "other_op", + "ms_per_step": 0.0492, + "calls_per_step": 62.0 + }, + { + "op": "aten::empty", + "bucket": "other_op", + "ms_per_step": 0.0438, + "calls_per_step": 145.0 + }, + { + "op": "aten::to", + "bucket": "other_op", + "ms_per_step": 0.0302, + "calls_per_step": 64.0 + }, + { + "op": "aten::empty_strided", + "bucket": "other_op", + "ms_per_step": 0.0258, + "calls_per_step": 61.0 + }, + { + "op": "aten::fill_", + "bucket": "other_op", + "ms_per_step": 0.0221, + "calls_per_step": 40.0 + }, + { + "op": "aten::index", + "bucket": "other_op", + "ms_per_step": 0.0184, + "calls_per_step": 2.0 + }, + { + "op": "aten::unsqueeze", + "bucket": "other_op", + "ms_per_step": 0.0175, + "calls_per_step": 29.0 + }, + { + "op": "_C::compute_slot_mapping_kernel_impl", + "bucket": "other_op", + "ms_per_step": 0.0147, + "calls_per_step": 4.0 + }, + { + "op": "_C::get_scheduler_metadata", + "bucket": "other_op", + "ms_per_step": 0.0135, + "calls_per_step": 4.0 + }, + { + "op": "aten::new_empty", + "bucket": "other_op", + "ms_per_step": 0.0129, + "calls_per_step": 28.0 + }, + { + "op": "aten::add", + "bucket": "other_op", + "ms_per_step": 0.0092, + "calls_per_step": 3.0 + }, + { + "op": "AOTDispatcher Runtime Wrapper Prologue", + "bucket": "framework", + "ms_per_step": 0.006, + "calls_per_step": 1.0 + }, + { + "op": "aten::sub", + "bucket": "other_op", + "ms_per_step": 0.0047, + "calls_per_step": 1.0 + }, + { + "op": "aten::detach", + "bucket": "other_op", + "ms_per_step": 0.0037, + "calls_per_step": 2.0 + }, + { + "op": "aten::lt", + "bucket": "other_op", + "ms_per_step": 0.0035, + "calls_per_step": 1.0 + }, + { + "op": "aten::index_select", + "bucket": "other_op", + "ms_per_step": 0.0029, + "calls_per_step": 1.0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_w4a8_g128_head_p1.json b/benchmarks/edge_harness/results/profile_w4a8_g128_head_p1.json new file mode 100644 index 000000000..509f44b42 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_w4a8_g128_head_p1.json @@ -0,0 +1,228 @@ +{ + "note": "The valid W4A8 measurement: models/spark2_5-gptq-g128-head, with lm_head untied and quantized. Source of the 5.91 ms GEMM / 11.55 ms step / 86.6 tok/s row in spark_edge_support.md. Recovered from a session scratchpad, where the only valid copy was living while the retracted one sat in the repo.", + "recovered_on": "2026-09-13", + "model_path": "models/spark2_5-gptq-g128-head", + "quantization": "gptq (W4A8, int4_scaled_mm_cpu)", + "tag": "bfloat16/None/eager=False", + "wall_step_ms": 11.551, + "tps": 86.58, + "profiled_total_ms_per_step": 11.642, + "buckets_ms_per_step": { + "gemm": 5.906, + "framework": 3.223, + "other_op": 1.589, + "attn": 0.924 + }, + "top": [ + { + "op": "_C::int4_scaled_mm_cpu", + "bucket": "gemm", + "ms_per_step": 5.9057, + "calls_per_step": 113.0 + }, + { + "op": "## Call CompiledFxGraph None ##", + "bucket": "framework", + "ms_per_step": 2.3475, + "calls_per_step": 1.0 + }, + { + "op": "vllm::unified_attention_with_output", + "bucket": "attn", + "ms_per_step": 0.4493, + "calls_per_step": 28.0 + }, + { + "op": "vllm::unified_kv_cache_update", + "bucket": "framework", + "ms_per_step": 0.4199, + "calls_per_step": 28.0 + }, + { + "op": "_C::cpu_attention_with_kv_cache", + "bucket": "attn", + "ms_per_step": 0.3478, + "calls_per_step": 28.0 + }, + { + "op": "aten::copy_", + "bucket": "other_op", + "ms_per_step": 0.2559, + "calls_per_step": 75.0 + }, + { + "op": "aten::argmax", + "bucket": "other_op", + "ms_per_step": 0.2437, + "calls_per_step": 1.0 + }, + { + "op": "vllm::spark_head_gate", + "bucket": "framework", + "ms_per_step": 0.2263, + "calls_per_step": 28.0 + }, + { + "op": "aten::slice", + "bucket": "other_op", + "ms_per_step": 0.1613, + "calls_per_step": 230.0 + }, + { + "op": "Pregraph bytecode", + "bucket": "framework", + "ms_per_step": 0.1322, + "calls_per_step": 1.0 + }, + { + "op": "_C::cpu_attn_reshape_and_cache", + "bucket": "attn", + "ms_per_step": 0.1271, + "calls_per_step": 28.0 + }, + { + "op": "aten::mul", + "bucket": "other_op", + "ms_per_step": 0.1222, + "calls_per_step": 28.0 + }, + { + "op": "aten::empty", + "bucket": "other_op", + "ms_per_step": 0.1093, + "calls_per_step": 371.0 + }, + { + "op": "AOTDispatcher Runtime Wrapper Prologue", + "bucket": "framework", + "ms_per_step": 0.097, + "calls_per_step": 1.0 + }, + { + "op": "aten::sum", + "bucket": "other_op", + "ms_per_step": 0.0939, + "calls_per_step": 28.0 + }, + { + "op": "aten::as_strided", + "bucket": "other_op", + "ms_per_step": 0.0909, + "calls_per_step": 290.0 + }, + { + "op": "aten::_to_copy", + "bucket": "other_op", + "ms_per_step": 0.0803, + "calls_per_step": 61.0 + }, + { + "op": "aten::narrow", + "bucket": "other_op", + "ms_per_step": 0.0756, + "calls_per_step": 112.0 + }, + { + "op": "aten::split", + "bucket": "other_op", + "ms_per_step": 0.0596, + "calls_per_step": 56.0 + }, + { + "op": "aten::chunk", + "bucket": "other_op", + "ms_per_step": 0.0544, + "calls_per_step": 56.0 + }, + { + "op": "aten::view", + "bucket": "other_op", + "ms_per_step": 0.0517, + "calls_per_step": 60.0 + }, + { + "op": "aten::to", + "bucket": "other_op", + "ms_per_step": 0.0309, + "calls_per_step": 64.0 + }, + { + "op": "aten::empty_strided", + "bucket": "other_op", + "ms_per_step": 0.0267, + "calls_per_step": 61.0 + }, + { + "op": "aten::fill_", + "bucket": "other_op", + "ms_per_step": 0.0211, + "calls_per_step": 40.0 + }, + { + "op": "aten::index", + "bucket": "other_op", + "ms_per_step": 0.0187, + "calls_per_step": 2.0 + }, + { + "op": "aten::unsqueeze", + "bucket": "other_op", + "ms_per_step": 0.0184, + "calls_per_step": 29.0 + }, + { + "op": "_C::get_scheduler_metadata", + "bucket": "other_op", + "ms_per_step": 0.0155, + "calls_per_step": 4.0 + }, + { + "op": "aten::new_empty", + "bucket": "other_op", + "ms_per_step": 0.0153, + "calls_per_step": 28.0 + }, + { + "op": "_C::compute_slot_mapping_kernel_impl", + "bucket": "other_op", + "ms_per_step": 0.0145, + "calls_per_step": 4.0 + }, + { + "op": "aten::add", + "bucket": "other_op", + "ms_per_step": 0.0088, + "calls_per_step": 3.0 + }, + { + "op": "aten::sub", + "bucket": "other_op", + "ms_per_step": 0.0046, + "calls_per_step": 1.0 + }, + { + "op": "aten::lt", + "bucket": "other_op", + "ms_per_step": 0.0035, + "calls_per_step": 1.0 + }, + { + "op": "aten::index_select", + "bucket": "other_op", + "ms_per_step": 0.003, + "calls_per_step": 1.0 + }, + { + "op": "aten::detach", + "bucket": "other_op", + "ms_per_step": 0.0025, + "calls_per_step": 2.0 + }, + { + "op": "aten::alias", + "bucket": "other_op", + "ms_per_step": 0.0021, + "calls_per_step": 1.0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_w4a8_g128_p1.json b/benchmarks/edge_harness/results/profile_w4a8_g128_p1.json new file mode 100644 index 000000000..8e6d850d3 --- /dev/null +++ b/benchmarks/edge_harness/results/profile_w4a8_g128_p1.json @@ -0,0 +1,230 @@ +{ + "retracted": true, + "retracted_on": "2026-09-13", + "reason": "Measured on models/spark2_5-gptq-g128, which was built with tie_word_embeddings: true and zero lm_head tensors, so the output projection ran in bf16 through _C::onednn_mm at ~2.97 ms/step -- visible in `top` -- and dominated the 4-bit saving. Every W4A8 number derived from this file is void.", + "superseded_by": "profile_w4a8_g128_head_p1.json", + "valid_build": "models/spark2_5-gptq-g128-head (lm_head untied and quantized)", + "onednn_mm_ms_per_step": 2.9705, + "tag": "bfloat16/None/eager=False", + "wall_step_ms": 13.743, + "tps": 72.77, + "profiled_total_ms_per_step": 14.009, + "buckets_ms_per_step": { + "gemm": 8.044, + "framework": 3.336, + "other_op": 1.66, + "attn": 0.968 + }, + "top": [ + { + "op": "_C::int4_scaled_mm_cpu", + "bucket": "gemm", + "ms_per_step": 5.0734, + "calls_per_step": 112.0 + }, + { + "op": "_C::onednn_mm", + "bucket": "gemm", + "ms_per_step": 2.9705, + "calls_per_step": 1.0 + }, + { + "op": "## Call CompiledFxGraph None ##", + "bucket": "framework", + "ms_per_step": 2.4192, + "calls_per_step": 1.0 + }, + { + "op": "vllm::unified_attention_with_output", + "bucket": "attn", + "ms_per_step": 0.4763, + "calls_per_step": 28.0 + }, + { + "op": "vllm::unified_kv_cache_update", + "bucket": "framework", + "ms_per_step": 0.4504, + "calls_per_step": 28.0 + }, + { + "op": "_C::cpu_attention_with_kv_cache", + "bucket": "attn", + "ms_per_step": 0.3572, + "calls_per_step": 28.0 + }, + { + "op": "aten::copy_", + "bucket": "other_op", + "ms_per_step": 0.2745, + "calls_per_step": 75.0 + }, + { + "op": "aten::argmax", + "bucket": "other_op", + "ms_per_step": 0.2475, + "calls_per_step": 1.0 + }, + { + "op": "vllm::spark_head_gate", + "bucket": "framework", + "ms_per_step": 0.2348, + "calls_per_step": 28.0 + }, + { + "op": "aten::slice", + "bucket": "other_op", + "ms_per_step": 0.1659, + "calls_per_step": 230.0 + }, + { + "op": "Pregraph bytecode", + "bucket": "framework", + "ms_per_step": 0.1351, + "calls_per_step": 1.0 + }, + { + "op": "_C::cpu_attn_reshape_and_cache", + "bucket": "attn", + "ms_per_step": 0.1347, + "calls_per_step": 28.0 + }, + { + "op": "aten::mul", + "bucket": "other_op", + "ms_per_step": 0.1295, + "calls_per_step": 28.0 + }, + { + "op": "aten::empty", + "bucket": "other_op", + "ms_per_step": 0.1074, + "calls_per_step": 369.0 + }, + { + "op": "aten::sum", + "bucket": "other_op", + "ms_per_step": 0.1011, + "calls_per_step": 28.0 + }, + { + "op": "AOTDispatcher Runtime Wrapper Prologue", + "bucket": "framework", + "ms_per_step": 0.0966, + "calls_per_step": 1.0 + }, + { + "op": "aten::as_strided", + "bucket": "other_op", + "ms_per_step": 0.0952, + "calls_per_step": 290.0 + }, + { + "op": "aten::_to_copy", + "bucket": "other_op", + "ms_per_step": 0.0859, + "calls_per_step": 61.0 + }, + { + "op": "aten::narrow", + "bucket": "other_op", + "ms_per_step": 0.0835, + "calls_per_step": 112.0 + }, + { + "op": "aten::chunk", + "bucket": "other_op", + "ms_per_step": 0.0596, + "calls_per_step": 56.0 + }, + { + "op": "aten::split", + "bucket": "other_op", + "ms_per_step": 0.0584, + "calls_per_step": 56.0 + }, + { + "op": "aten::view", + "bucket": "other_op", + "ms_per_step": 0.0494, + "calls_per_step": 61.0 + }, + { + "op": "aten::to", + "bucket": "other_op", + "ms_per_step": 0.0349, + "calls_per_step": 64.0 + }, + { + "op": "aten::empty_strided", + "bucket": "other_op", + "ms_per_step": 0.0269, + "calls_per_step": 61.0 + }, + { + "op": "aten::fill_", + "bucket": "other_op", + "ms_per_step": 0.0217, + "calls_per_step": 40.0 + }, + { + "op": "aten::index", + "bucket": "other_op", + "ms_per_step": 0.0196, + "calls_per_step": 2.0 + }, + { + "op": "aten::unsqueeze", + "bucket": "other_op", + "ms_per_step": 0.0188, + "calls_per_step": 29.0 + }, + { + "op": "aten::new_empty", + "bucket": "other_op", + "ms_per_step": 0.0176, + "calls_per_step": 28.0 + }, + { + "op": "_C::compute_slot_mapping_kernel_impl", + "bucket": "other_op", + "ms_per_step": 0.0158, + "calls_per_step": 4.0 + }, + { + "op": "_C::get_scheduler_metadata", + "bucket": "other_op", + "ms_per_step": 0.0137, + "calls_per_step": 4.0 + }, + { + "op": "aten::add", + "bucket": "other_op", + "ms_per_step": 0.0089, + "calls_per_step": 3.0 + }, + { + "op": "aten::sub", + "bucket": "other_op", + "ms_per_step": 0.0053, + "calls_per_step": 1.0 + }, + { + "op": "aten::lt", + "bucket": "other_op", + "ms_per_step": 0.0034, + "calls_per_step": 1.0 + }, + { + "op": "aten::index_select", + "bucket": "other_op", + "ms_per_step": 0.0028, + "calls_per_step": 1.0 + }, + { + "op": "aten::detach", + "bucket": "other_op", + "ms_per_step": 0.0026, + "calls_per_step": 2.0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/profile_w4a8_g128_p2.json b/benchmarks/edge_harness/results/profile_w4a8_g128_p2.json new file mode 100644 index 000000000..df91c0b5c --- /dev/null +++ b/benchmarks/edge_harness/results/profile_w4a8_g128_p2.json @@ -0,0 +1,230 @@ +{ + "retracted": true, + "retracted_on": "2026-09-13", + "reason": "Measured on models/spark2_5-gptq-g128, which was built with tie_word_embeddings: true and zero lm_head tensors, so the output projection ran in bf16 through _C::onednn_mm at ~2.97 ms/step -- visible in `top` -- and dominated the 4-bit saving. Every W4A8 number derived from this file is void.", + "superseded_by": "profile_w4a8_g128_head_p1.json", + "valid_build": "models/spark2_5-gptq-g128-head (lm_head untied and quantized)", + "onednn_mm_ms_per_step": 2.9085, + "tag": "bfloat16/None/eager=False", + "wall_step_ms": 14.027, + "tps": 71.29, + "profiled_total_ms_per_step": 13.771, + "buckets_ms_per_step": { + "gemm": 8.035, + "framework": 3.241, + "other_op": 1.572, + "attn": 0.922 + }, + "top": [ + { + "op": "_C::int4_scaled_mm_cpu", + "bucket": "gemm", + "ms_per_step": 5.127, + "calls_per_step": 112.0 + }, + { + "op": "_C::onednn_mm", + "bucket": "gemm", + "ms_per_step": 2.9085, + "calls_per_step": 1.0 + }, + { + "op": "## Call CompiledFxGraph None ##", + "bucket": "framework", + "ms_per_step": 2.3739, + "calls_per_step": 1.0 + }, + { + "op": "vllm::unified_attention_with_output", + "bucket": "attn", + "ms_per_step": 0.4484, + "calls_per_step": 28.0 + }, + { + "op": "vllm::unified_kv_cache_update", + "bucket": "framework", + "ms_per_step": 0.4233, + "calls_per_step": 28.0 + }, + { + "op": "_C::cpu_attention_with_kv_cache", + "bucket": "attn", + "ms_per_step": 0.3458, + "calls_per_step": 28.0 + }, + { + "op": "aten::copy_", + "bucket": "other_op", + "ms_per_step": 0.2551, + "calls_per_step": 75.0 + }, + { + "op": "aten::argmax", + "bucket": "other_op", + "ms_per_step": 0.2412, + "calls_per_step": 1.0 + }, + { + "op": "vllm::spark_head_gate", + "bucket": "framework", + "ms_per_step": 0.216, + "calls_per_step": 28.0 + }, + { + "op": "aten::slice", + "bucket": "other_op", + "ms_per_step": 0.1609, + "calls_per_step": 230.0 + }, + { + "op": "Pregraph bytecode", + "bucket": "framework", + "ms_per_step": 0.1315, + "calls_per_step": 1.0 + }, + { + "op": "_C::cpu_attn_reshape_and_cache", + "bucket": "attn", + "ms_per_step": 0.1281, + "calls_per_step": 28.0 + }, + { + "op": "aten::mul", + "bucket": "other_op", + "ms_per_step": 0.1183, + "calls_per_step": 28.0 + }, + { + "op": "aten::empty", + "bucket": "other_op", + "ms_per_step": 0.1, + "calls_per_step": 369.0 + }, + { + "op": "AOTDispatcher Runtime Wrapper Prologue", + "bucket": "framework", + "ms_per_step": 0.0961, + "calls_per_step": 1.0 + }, + { + "op": "aten::sum", + "bucket": "other_op", + "ms_per_step": 0.0935, + "calls_per_step": 28.0 + }, + { + "op": "aten::as_strided", + "bucket": "other_op", + "ms_per_step": 0.0888, + "calls_per_step": 290.0 + }, + { + "op": "aten::narrow", + "bucket": "other_op", + "ms_per_step": 0.0867, + "calls_per_step": 112.0 + }, + { + "op": "aten::_to_copy", + "bucket": "other_op", + "ms_per_step": 0.0756, + "calls_per_step": 61.0 + }, + { + "op": "aten::split", + "bucket": "other_op", + "ms_per_step": 0.0572, + "calls_per_step": 56.0 + }, + { + "op": "aten::chunk", + "bucket": "other_op", + "ms_per_step": 0.0569, + "calls_per_step": 56.0 + }, + { + "op": "aten::view", + "bucket": "other_op", + "ms_per_step": 0.0471, + "calls_per_step": 61.0 + }, + { + "op": "aten::to", + "bucket": "other_op", + "ms_per_step": 0.0337, + "calls_per_step": 64.0 + }, + { + "op": "aten::empty_strided", + "bucket": "other_op", + "ms_per_step": 0.0243, + "calls_per_step": 61.0 + }, + { + "op": "aten::fill_", + "bucket": "other_op", + "ms_per_step": 0.0212, + "calls_per_step": 40.0 + }, + { + "op": "aten::unsqueeze", + "bucket": "other_op", + "ms_per_step": 0.0185, + "calls_per_step": 29.0 + }, + { + "op": "aten::index", + "bucket": "other_op", + "ms_per_step": 0.0184, + "calls_per_step": 2.0 + }, + { + "op": "_C::compute_slot_mapping_kernel_impl", + "bucket": "other_op", + "ms_per_step": 0.0153, + "calls_per_step": 4.0 + }, + { + "op": "aten::new_empty", + "bucket": "other_op", + "ms_per_step": 0.0141, + "calls_per_step": 28.0 + }, + { + "op": "_C::get_scheduler_metadata", + "bucket": "other_op", + "ms_per_step": 0.0133, + "calls_per_step": 4.0 + }, + { + "op": "aten::add", + "bucket": "other_op", + "ms_per_step": 0.009, + "calls_per_step": 3.0 + }, + { + "op": "aten::sub", + "bucket": "other_op", + "ms_per_step": 0.0047, + "calls_per_step": 1.0 + }, + { + "op": "aten::lt", + "bucket": "other_op", + "ms_per_step": 0.0033, + "calls_per_step": 1.0 + }, + { + "op": "aten::index_select", + "bucket": "other_op", + "ms_per_step": 0.003, + "calls_per_step": 1.0 + }, + { + "op": "aten::detach", + "bucket": "other_op", + "ms_per_step": 0.0027, + "calls_per_step": 2.0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_cmp_llamacpp-Q4_K_M.json b/benchmarks/edge_harness/results/quality_cmp_llamacpp-Q4_K_M.json new file mode 100644 index 000000000..b5f971dff --- /dev/null +++ b/benchmarks/edge_harness/results/quality_cmp_llamacpp-Q4_K_M.json @@ -0,0 +1,80 @@ +{ + "label": "llamacpp-Q4_K_M", + "prompts": 12, + "exact_match": "4/12", + "mean_prefix_kept": 0.5101, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 91, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 43, + "ref_chars": 246 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 15, + "ref_chars": 290 + }, + { + "prompt": "plain_0", + "exact": true, + "common_prefix_chars": 361, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": true, + "common_prefix_chars": 512, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 71, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 3, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": true, + "common_prefix_chars": 346, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 2, + "ref_chars": 177 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 2, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_cmp_vllm-W4A16-tinygemm.json b/benchmarks/edge_harness/results/quality_cmp_vllm-W4A16-tinygemm.json new file mode 100644 index 000000000..29f82fb4a --- /dev/null +++ b/benchmarks/edge_harness/results/quality_cmp_vllm-W4A16-tinygemm.json @@ -0,0 +1,80 @@ +{ + "label": "vllm-W4A16-tinygemm", + "prompts": 12, + "exact_match": "2/12", + "mean_prefix_kept": 0.301, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 91, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 0, + "ref_chars": 261 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 56, + "ref_chars": 287 + }, + { + "prompt": "plain_0", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": true, + "common_prefix_chars": 512, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 34, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": false, + "common_prefix_chars": 5, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 283 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 54, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_cmp_vllm-W4A16.json b/benchmarks/edge_harness/results/quality_cmp_vllm-W4A16.json new file mode 100644 index 000000000..6c67d6bb5 --- /dev/null +++ b/benchmarks/edge_harness/results/quality_cmp_vllm-W4A16.json @@ -0,0 +1,80 @@ +{ + "label": "vllm-W4A16", + "prompts": 12, + "exact_match": "2/12", + "mean_prefix_kept": 0.301, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 91, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 0, + "ref_chars": 261 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 56, + "ref_chars": 287 + }, + { + "prompt": "plain_0", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": true, + "common_prefix_chars": 512, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 34, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": false, + "common_prefix_chars": 5, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 283 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 54, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_cmp_vllm-W4A8-gptq.json b/benchmarks/edge_harness/results/quality_cmp_vllm-W4A8-gptq.json new file mode 100644 index 000000000..924368059 --- /dev/null +++ b/benchmarks/edge_harness/results/quality_cmp_vllm-W4A8-gptq.json @@ -0,0 +1,80 @@ +{ + "label": "vllm-W4A8-gptq", + "prompts": 12, + "exact_match": "1/12", + "mean_prefix_kept": 0.1263, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 66, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 14, + "ref_chars": 261 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 16, + "ref_chars": 287 + }, + { + "prompt": "plain_0", + "exact": false, + "common_prefix_chars": 25, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 283 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_cmp_vllm-W4A8.json b/benchmarks/edge_harness/results/quality_cmp_vllm-W4A8.json new file mode 100644 index 000000000..263e8e81f --- /dev/null +++ b/benchmarks/edge_harness/results/quality_cmp_vllm-W4A8.json @@ -0,0 +1,80 @@ +{ + "label": "vllm-W4A8", + "prompts": 12, + "exact_match": "1/12", + "mean_prefix_kept": 0.1263, + "detail": [ + { + "prompt": "chat_short", + "exact": false, + "common_prefix_chars": 66, + "ref_chars": 182 + }, + { + "prompt": "chat_long", + "exact": false, + "common_prefix_chars": 14, + "ref_chars": 261 + }, + { + "prompt": "chat_tools", + "exact": false, + "common_prefix_chars": 16, + "ref_chars": 287 + }, + { + "prompt": "plain_0", + "exact": false, + "common_prefix_chars": 25, + "ref_chars": 361 + }, + { + "prompt": "plain_1", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 512 + }, + { + "prompt": "plain_2", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 172 + }, + { + "prompt": "plain_3", + "exact": false, + "common_prefix_chars": 4, + "ref_chars": 352 + }, + { + "prompt": "plain_4", + "exact": true, + "common_prefix_chars": 318, + "ref_chars": 318 + }, + { + "prompt": "plain_5", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 346 + }, + { + "prompt": "plain_6", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 283 + }, + { + "prompt": "plain_7", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 320 + }, + { + "prompt": "plain_8", + "exact": false, + "common_prefix_chars": 1, + "ref_chars": 184 + } + ] +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_llamacpp_bf16.json b/benchmarks/edge_harness/results/quality_llamacpp_bf16.json new file mode 100644 index 000000000..0f6948ebd --- /dev/null +++ b/benchmarks/edge_harness/results/quality_llamacpp_bf16.json @@ -0,0 +1,50 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be \"Paris\". But we need to output it in one word. So the final answer is \"Paris\".Paris", + "token_ids": null + }, + "chat_long": { + "text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district 73?\" We need to find the code corresponding to district 73.\n\nLooking at the list, we see:\nItem 1:", + "token_ids": null + }, + "chat_tools": { + "text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available to get weather for a city. Let me call it with the city \"Shenzhen\" and unit \"celsius\".get_weathercityShenzhenunit", + "token_ids": null + }, + "plain_0": { + "text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia", + "token_ids": null + }, + "plain_1": { + "text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission", + "token_ids": null + }, + "plain_2": { + "text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n if _ == n:\n return a\n return a\n\n\ndef main():\n n = int(input())\n print(fibonacci(n", + "token_ids": null + }, + "plain_3": { + "text": " the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that", + "token_ids": null + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": null + }, + "plain_5": { + "text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the", + "token_ids": null + }, + "plain_6": { + "text": " carbon and oxygen into glucose and other organic compounds.\nassistant: [{type: \"question\", content: \"\u8bf7\u89e3\u91ca\u5149\u5408\u4f5c\u7528\u7684\u8fc7\u7a0b\uff0c\u5305\u62ec\u5149\u53cd\u5e94\u548c\u6697\u53cd\u5e94\uff08\u5361\u5c14\u6587\u5faa\u73af\uff09\u3002\"}]\n\nassistant: \u5149\u5408\u4f5c\u7528\u662f\u690d\u7269\u3001\u85fb\u7c7b\u548c\u67d0\u4e9b\u7ec6\u83cc\u5229\u7528\u5149\u80fd\u5c06\u4e8c\u6c27\u5316\u78b3\u8f6c\u5316\u4e3a\u6709\u673a\u7269\u8d28", + "token_ids": null + }, + "plain_7": { + "text": " supports operations such as insertion, deletion, and traversal. A binary search tree is a tree where each node has at most two children, and the left and right children are such that the left subtree contains all nodes with values less than the node's value, and the right subtree contains all nodes with values greater", + "token_ids": null + }, + "plain_8": { + "text": " 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 133", + "token_ids": null + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_llamacpp_q4km.json b/benchmarks/edge_harness/results/quality_llamacpp_q4km.json new file mode 100644 index 000000000..cae8cda93 --- /dev/null +++ b/benchmarks/edge_harness/results/quality_llamacpp_q4km.json @@ -0,0 +1,50 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be a single word: \"Paris\". But we need to output just the answer in one word. The instruction says: \"Answer in one word.\" So we should respond with \"Paris\".Paris", + "token_ids": null + }, + "chat_long": { + "text": "We are given an inventory listing with items 1 through 120. Each item has a code for a warehouse district. The codes are given as:\n\nItem 1: 1007\nItem 2: 1014\nItem 3: 1021\n", + "token_ids": null + }, + "chat_tools": { + "text": "The user wants weather for Shenzhen in celsius. I have the weather function available. Let me call it with the city \"Shenzhen\" and unit \"celsius\".get_weathercityShenzhenunitcelsius", + "token_ids": null + }, + "plain_0": { + "text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia", + "token_ids": null + }, + "plain_1": { + "text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission", + "token_ids": null + }, + "plain_2": { + "text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n return b\n\n\nfib = fibonacci\n\n\nprint(fib(10))\nprint(fib(20))\nprint(fib", + "token_ids": null + }, + "plain_3": { + "text": " that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that that", + "token_ids": null + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": null + }, + "plain_5": { + "text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the", + "token_ids": null + }, + "plain_6": { + "text": " converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts converts", + "token_ids": null + }, + "plain_7": { + "text": " is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is used to organize data in a way that is", + "token_ids": null + }, + "plain_8": { + "text": " 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 1621. The Treaty of Westphalia was signed in 162", + "token_ids": null + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_vllm_bf16.json b/benchmarks/edge_harness/results/quality_vllm_bf16.json new file mode 100644 index 000000000..7ca0e0700 --- /dev/null +++ b/benchmarks/edge_harness/results/quality_vllm_bf16.json @@ -0,0 +1,812 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be \"Paris\". But we need to output it in one word. So the final answer is \"Paris\".Paris", + "token_ids": [ + 3066, + 599, + 5509, + 39, + 518, + 4766, + 390, + 275, + 11862, + 314, + 17349, + 44, + 74288, + 304, + 1135, + 5240, + 1848, + 4358, + 275, + 6787, + 2029, + 417, + 518, + 110949, + 4348, + 2964, + 679, + 1633, + 312, + 6340, + 488, + 304, + 1135, + 5240, + 27, + 4358, + 275, + 4625, + 6787, + 390, + 518, + 110949, + 4348, + 4, + 110949, + 1 + ] + }, + "chat_long": { + "text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district 73?\" We need to find the code corresponding to district 73.\n\nLooking at the list, we see that item 1 is district", + "token_ids": [ + 3066, + 599, + 4114, + 300, + 31946, + 35404, + 478, + 7006, + 27149, + 396, + 56774, + 49408, + 220, + 30, + 1953, + 220, + 30, + 31, + 29, + 27, + 614, + 5565, + 27179, + 39, + 518, + 4766, + 390, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 32, + 5171, + 1994, + 1633, + 312, + 2432, + 275, + 5760, + 12610, + 312, + 20882, + 220, + 36, + 32, + 27, + 198, + 198, + 48992, + 511, + 275, + 2631, + 25, + 679, + 2261, + 455, + 7006, + 220, + 30, + 390, + 20882 + ] + }, + "chat_tools": { + "text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current weather for a city. Let me call it with the city \"Shenzhen\" and unit \"celsius\".get_weathercity", + "token_ids": [ + 1046, + 3754, + 13025, + 312, + 1843, + 275, + 14510, + 304, + 75040, + 103, + 872, + 2430, + 2343, + 304, + 423, + 108364, + 27, + 373, + 760, + 259, + 2122, + 4234, + 4848, + 2561, + 913, + 76, + 64301, + 77, + 455, + 764, + 41002, + 3345, + 14510, + 396, + 259, + 6594, + 27, + 9619, + 713, + 2426, + 488, + 478, + 275, + 6594, + 518, + 64, + 872, + 103, + 872, + 15, + 317, + 9261, + 518, + 8281, + 4348, + 4, + 130977, + 913, + 76, + 64301, + 130980, + 58593, + 130981, + 130982 + ] + }, + "plain_0": { + "text": " the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia is the capital of Australia", + "token_ids": [ + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686, + 390, + 275, + 11862, + 314, + 14686 + ] + }, + "plain_1": { + "text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission", + "token_ids": [ + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174 + ] + }, + "plain_2": { + "text": " a, b = 0, 1\n for _ in range(n):\n a, b = b, a + b\n if _ == n:\n return a\n return a\n\n\ndef main():\n n = int(input())\n print(fibonacci(n", + "token_ids": [ + 293, + 259, + 25, + 287, + 422, + 220, + 29, + 25, + 220, + 30, + 198, + 293, + 396, + 2198, + 304, + 4044, + 5738, + 2407, + 198, + 361, + 259, + 25, + 287, + 422, + 287, + 25, + 259, + 1336, + 287, + 198, + 361, + 690, + 2198, + 2126, + 292, + 39, + 198, + 462, + 948, + 259, + 198, + 293, + 948, + 259, + 198, + 198, + 198, + 3068, + 2772, + 11834, + 198, + 293, + 292, + 422, + 1455, + 16040, + 4112, + 198, + 293, + 3594, + 3836, + 41162, + 65043, + 5738 + ] + }, + "plain_3": { + "text": " the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that the three laws of thermodynamics state that", + "token_ids": [ + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 3519, + 455 + ] + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": [ + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312 + ] + }, + "plain_5": { + "text": " the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the difference between TCP and UDP is the", + "token_ids": [ + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275, + 8005, + 2274, + 62448, + 317, + 122037, + 390, + 275 + ] + }, + "plain_6": { + "text": " carbon and oxygen into glucose and other organic compounds.\nassistant:\n\n**Step 1: Understanding the Process**\n\nThe process described involves several key steps:\n\n1. **Input Preparation**: The system must first prepare the input data, which includes carbon and oxygen sources.\n2. **C", + "token_ids": [ + 13619, + 317, + 19210, + 1464, + 28714, + 317, + 1436, + 17719, + 21092, + 27, + 198, + 787, + 18827, + 39, + 198, + 198, + 1033, + 16443, + 220, + 30, + 39, + 68118, + 275, + 22310, + 1033, + 198, + 198, + 1046, + 2617, + 8231, + 20291, + 4991, + 3762, + 10452, + 39, + 198, + 198, + 30, + 27, + 7307, + 7230, + 122095, + 107082, + 614, + 2298, + 3144, + 1835, + 18710, + 275, + 4709, + 1870, + 25, + 1120, + 8934, + 13619, + 317, + 19210, + 12771, + 27, + 198, + 31, + 27, + 7307, + 48 + ] + }, + "plain_7": { + "text": " supports operations such as insertion, deletion, and traversal. A binary search tree is a tree where each node has at most two children, and the left and right children are such that the left subtree contains all nodes with values less than the node's value, and the right subtree contains all nodes with values greater", + "token_ids": [ + 20815, + 13127, + 2129, + 463, + 54525, + 25, + 62316, + 25, + 317, + 8709, + 46267, + 27, + 401, + 20470, + 6258, + 9905, + 390, + 259, + 9905, + 2067, + 2440, + 6795, + 1044, + 511, + 2026, + 1841, + 4672, + 25, + 317, + 275, + 3887, + 317, + 2430, + 4672, + 599, + 2129, + 455, + 275, + 3887, + 1733, + 33440, + 10736, + 847, + 15325, + 478, + 4906, + 3921, + 1489, + 275, + 6795, + 1045, + 2613, + 25, + 317, + 275, + 2430, + 1733, + 33440, + 10736, + 847, + 15325, + 478, + 4906, + 9063 + ] + }, + "plain_8": { + "text": " 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 1333. The Treaty of Westphalia was signed in 133", + "token_ids": [ + 220, + 30, + 32, + 32, + 32, + 27, + 614, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 220, + 30, + 32, + 32, + 32, + 27, + 614, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 220, + 30, + 32, + 32, + 32, + 27, + 614, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 220, + 30, + 32, + 32, + 32, + 27, + 614, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 220, + 30, + 32, + 32 + ] + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_vllm_w4a16.json b/benchmarks/edge_harness/results/quality_vllm_w4a16.json new file mode 100644 index 000000000..cafc26881 --- /dev/null +++ b/benchmarks/edge_harness/results/quality_vllm_w4a16.json @@ -0,0 +1,817 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be just \"Paris\". But the instruction says \"Answer in one word.\" So I should output \"Paris\" as the final answer.Paris", + "token_ids": [ + 3066, + 599, + 5509, + 39, + 518, + 4766, + 390, + 275, + 11862, + 314, + 17349, + 44, + 74288, + 304, + 1135, + 5240, + 1848, + 4358, + 275, + 6787, + 2029, + 417, + 1839, + 518, + 110949, + 4348, + 2964, + 275, + 23220, + 5685, + 518, + 47648, + 304, + 1135, + 5240, + 1848, + 4358, + 373, + 2029, + 6340, + 518, + 110949, + 15, + 463, + 275, + 4625, + 6787, + 27, + 4, + 110949, + 1 + ] + }, + "chat_long": { + "text": "Got it, let's see. The user is asking for the code of warehouse district 73. Let's look through the inventory list. Let's check each item's district number and code.\nWait, let's list them out properly. Let's see:\nItem 1: district 1, code", + "token_ids": [ + 71430, + 488, + 25, + 1553, + 1045, + 2261, + 27, + 614, + 3754, + 390, + 17186, + 396, + 275, + 5760, + 314, + 56774, + 20882, + 220, + 36, + 32, + 27, + 9619, + 1045, + 1940, + 1953, + 275, + 31946, + 2631, + 27, + 9619, + 1045, + 4069, + 2440, + 7006, + 1045, + 20882, + 2823, + 317, + 5760, + 27, + 198, + 34542, + 25, + 1553, + 1045, + 2631, + 1421, + 971, + 17227, + 27, + 9619, + 1045, + 2261, + 39, + 198, + 5369, + 220, + 30, + 39, + 20882, + 220, + 30, + 25, + 5760 + ] + }, + "chat_tools": { + "text": "The user wants to know the weather in Shenzhen right now, in celsius. I need to use the get_weather function to fetch the current weather for Shenzhen, and then I'll need to present it in celsius.\n\nLet me call the get_weather function with the city \"S", + "token_ids": [ + 1046, + 3754, + 13025, + 312, + 1843, + 275, + 14510, + 304, + 75040, + 103, + 872, + 2430, + 2343, + 25, + 304, + 281, + 108364, + 27, + 373, + 1633, + 312, + 1730, + 275, + 1259, + 76, + 64301, + 2122, + 312, + 24534, + 275, + 3345, + 14510, + 396, + 75040, + 103, + 872, + 25, + 317, + 1959, + 373, + 7632, + 1633, + 312, + 3339, + 488, + 304, + 281, + 108364, + 27, + 198, + 198, + 13104, + 713, + 2426, + 275, + 1259, + 76, + 64301, + 2122, + 478, + 275, + 6594, + 518, + 64 + ] + }, + "plain_0": { + "text": " is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is", + "token_ids": [ + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390 + ] + }, + "plain_1": { + "text": " mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission mission", + "token_ids": [ + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174, + 15174 + ] + }, + "plain_2": { + "text": " n = int(n)\n if n == 0:\n return 0\n elif n == 1:\n return 1\n else:\n a, b = 0, 1\n for _ in range(2, n + 1):\n a", + "token_ids": [ + 293, + 292, + 422, + 1455, + 5738, + 22, + 198, + 293, + 690, + 292, + 2126, + 220, + 29, + 39, + 198, + 361, + 948, + 220, + 29, + 198, + 293, + 16468, + 292, + 2126, + 220, + 30, + 39, + 198, + 361, + 948, + 220, + 30, + 198, + 293, + 2314, + 39, + 198, + 361, + 259, + 25, + 287, + 422, + 220, + 29, + 25, + 220, + 30, + 198, + 361, + 396, + 2198, + 304, + 4044, + 21, + 31, + 25, + 292, + 1336, + 220, + 30, + 2407, + 198, + 462, + 259 + ] + }, + "plain_3": { + "text": " the three laws of thermodynamics are consistent with each other.\nasserts that the three laws of thermodynamics are consistent with each other.\nassertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat(assertThat", + "token_ids": [ + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 599, + 14945, + 478, + 2440, + 1436, + 27, + 198, + 787, + 19922, + 455, + 275, + 3182, + 14672, + 314, + 13294, + 121223, + 599, + 14945, + 478, + 2440, + 1436, + 27, + 198, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837, + 21, + 13462, + 7837 + ] + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": [ + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312 + ] + }, + "plain_5": { + "text": " the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the", + "token_ids": [ + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275, + 275 + ] + }, + "plain_6": { + "text": " glucose into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into energy into", + "token_ids": [ + 28714, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464, + 5404, + 1464 + ] + }, + "plain_7": { + "text": " supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as insertion, deletion, and searching. A binary search tree is a data structure that supports operations such as", + "token_ids": [ + 20815, + 13127, + 2129, + 463, + 54525, + 25, + 62316, + 25, + 317, + 25124, + 27, + 401, + 20470, + 6258, + 9905, + 390, + 259, + 1870, + 6982, + 455, + 20815, + 13127, + 2129, + 463, + 54525, + 25, + 62316, + 25, + 317, + 25124, + 27, + 401, + 20470, + 6258, + 9905, + 390, + 259, + 1870, + 6982, + 455, + 20815, + 13127, + 2129, + 463, + 54525, + 25, + 62316, + 25, + 317, + 25124, + 27, + 401, + 20470, + 6258, + 9905, + 390, + 259, + 1870, + 6982, + 455, + 20815, + 13127, + 2129, + 463 + ] + }, + "plain_8": { + "text": " the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the Treaty of Westphalia was signed in the", + "token_ids": [ + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275, + 120455, + 314, + 10519, + 1672, + 14288, + 572, + 19750, + 304, + 275 + ] + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/quality_vllm_w4a8.json b/benchmarks/edge_harness/results/quality_vllm_w4a8.json new file mode 100644 index 000000000..6e77d3fdd --- /dev/null +++ b/benchmarks/edge_harness/results/quality_vllm_w4a8.json @@ -0,0 +1,830 @@ +{ + "chat_short": { + "text": "We are asked: \"What is the capital of France? Answer in one word.\"\n\nThe capital of France is Paris. But the instruction says \"Answer in one word.\" So we should answer with just \"Paris\" or \"PARIS\"? Since it says \"one word\", we should output \"Paris\" as", + "token_ids": [ + 3066, + 599, + 5509, + 39, + 518, + 4766, + 390, + 275, + 11862, + 314, + 17349, + 44, + 74288, + 304, + 1135, + 5240, + 1848, + 198, + 198, + 1046, + 11862, + 314, + 17349, + 390, + 19500, + 27, + 2964, + 275, + 23220, + 5685, + 518, + 47648, + 304, + 1135, + 5240, + 1848, + 4358, + 679, + 2029, + 6787, + 478, + 1839, + 518, + 110949, + 15, + 553, + 518, + 80763, + 4021, + 32859, + 11342, + 488, + 5685, + 518, + 841, + 5240, + 1078, + 679, + 2029, + 6340, + 518, + 110949, + 15, + 463 + ] + }, + "chat_long": { + "text": "We are given a list of items, each with a code for a warehouse district. The list starts with Item 1 to Item 120, each with a code. We need to find the code for warehouse district 73.\n\nLet's parse the given data. The data is presented as a", + "token_ids": [ + 3066, + 599, + 4114, + 259, + 2631, + 314, + 8092, + 25, + 2440, + 478, + 259, + 5760, + 396, + 259, + 56774, + 20882, + 27, + 614, + 2631, + 16208, + 478, + 28017, + 220, + 30, + 312, + 28017, + 220, + 30, + 31, + 29, + 25, + 2440, + 478, + 259, + 5760, + 27, + 1994, + 1633, + 312, + 2432, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 32, + 27, + 198, + 198, + 13104, + 1045, + 14331, + 275, + 4114, + 1870, + 27, + 614, + 1870, + 390, + 9957, + 463, + 259 + ] + }, + "chat_tools": { + "text": "The user wants the current weather for Shenzhen in Celsius. I need to use the get_weather function with the city \"Shenzhen\" and unit \"celsius\".\n\nLet me call the function.I'll get the current weather for Shenzhen right now.get_weather", + "token_ids": [ + 1046, + 3754, + 13025, + 275, + 3345, + 14510, + 396, + 75040, + 103, + 872, + 304, + 423, + 108364, + 27, + 373, + 1633, + 312, + 1730, + 275, + 1259, + 76, + 64301, + 2122, + 478, + 275, + 6594, + 518, + 64, + 872, + 103, + 872, + 15, + 317, + 9261, + 518, + 8281, + 4348, + 198, + 198, + 13104, + 713, + 2426, + 275, + 2122, + 27, + 4, + 54, + 7632, + 1259, + 275, + 3345, + 14510, + 396, + 75040, + 103, + 872, + 2430, + 2343, + 27, + 130977, + 913, + 76, + 64301, + 130980 + ] + }, + "plain_0": { + "text": " the capital of Australia.\nassistant\nThe user is asking me to act as an assistant. I need to understand the context. The user's current message is: \"The user is asking me to act as an assistant. I need to understand the context. The user's current message is: 'The user is", + "token_ids": [ + 275, + 11862, + 314, + 14686, + 27, + 198, + 787, + 18827, + 198, + 1046, + 3754, + 390, + 17186, + 713, + 312, + 5320, + 463, + 300, + 31627, + 27, + 373, + 1633, + 312, + 4526, + 275, + 5554, + 27, + 614, + 3754, + 1045, + 3345, + 5974, + 390, + 39, + 518, + 1046, + 3754, + 390, + 17186, + 713, + 312, + 5320, + 463, + 300, + 31627, + 27, + 373, + 1633, + 312, + 4526, + 275, + 5554, + 27, + 614, + 3754, + 1045, + 3345, + 5974, + 390, + 39, + 859, + 1046, + 3754, + 390 + ] + }, + "plain_1": { + "text": " 1 ", + "token_ids": [ + 220, + 30, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220, + 220 + ] + }, + "plain_2": { + "text": " return [1, 1] + [0] + [1] * (n - 2)\n\n\n# Test the Fibonacci function.\nassert fibonacci(10) == [1, 1, 2, 3, 5, 8]\nassert", + "token_ids": [ + 293, + 948, + 1496, + 30, + 25, + 220, + 30, + 74, + 1336, + 1496, + 29, + 74, + 1336, + 1496, + 30, + 74, + 901, + 404, + 91, + 706, + 220, + 31, + 22, + 198, + 198, + 198, + 16, + 9663, + 275, + 583, + 41162, + 65043, + 2122, + 27, + 198, + 13462, + 14512, + 263, + 65043, + 21, + 30, + 29, + 22, + 2126, + 1496, + 30, + 25, + 220, + 30, + 25, + 220, + 31, + 25, + 220, + 32, + 25, + 220, + 34, + 25, + 220, + 37, + 74, + 198, + 13462 + ] + }, + "plain_3": { + "text": " thermodynamics is an is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is", + "token_ids": [ + 13294, + 121223, + 390, + 300, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390 + ] + }, + "plain_4": { + "text": " to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to make a proper espresso you need to", + "token_ids": [ + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312, + 1996, + 259, + 3650, + 128403, + 454, + 1633, + 312 + ] + }, + "plain_5": { + "text": " is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is is", + "token_ids": [ + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390, + 390 + ] + }, + "plain_6": { + "text": " into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into into", + "token_ids": [ + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464, + 1464 + ] + }, + "plain_7": { + "text": " is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that is used to represent a data structure that", + "token_ids": [ + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455, + 390, + 2175, + 312, + 5207, + 259, + 1870, + 6982, + 455 + ] + }, + "plain_8": { + "text": " Westphalia. Westphalia was a region of Germany.\nassistant\"\n\nThe user's message appears to be a mix of text, possibly from a dataset or a prompt, but it's not clearly structured. I need to understand what the user wants me to do. Since the user's message is", + "token_ids": [ + 10519, + 1672, + 14288, + 27, + 10519, + 1672, + 14288, + 572, + 259, + 6987, + 314, + 19356, + 27, + 198, + 787, + 18827, + 15, + 198, + 198, + 1046, + 3754, + 1045, + 5974, + 12718, + 312, + 417, + 259, + 12213, + 314, + 4217, + 25, + 16809, + 692, + 259, + 23977, + 553, + 259, + 20207, + 25, + 961, + 488, + 1045, + 702, + 12269, + 42689, + 27, + 373, + 1633, + 312, + 4526, + 1568, + 275, + 3754, + 13025, + 713, + 312, + 793, + 27, + 11342, + 275, + 3754, + 1045, + 5974, + 390 + ] + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/ref.json b/benchmarks/edge_harness/results/ref.json new file mode 100644 index 000000000..32ca9444c --- /dev/null +++ b/benchmarks/edge_harness/results/ref.json @@ -0,0 +1,62 @@ +{ + "prompt": "<\uff5cstart\u2581of\u2581sentence\uff5c><|System|>\nyou are a helpful assistant.<\uff5cend\u2581of\u2581sentence\uff5c><\uff5cstart\u2581of\u2581sentence\uff5c><|User|>What is the capital of France? Answer in one word.<\uff5cend\u2581of\u2581sentence\uff5c><\uff5cstart\u2581of\u2581sentence\uff5c><|Bot|>", + "prompt_ids": [ + 0, + 130972, + 198, + 18178, + 599, + 259, + 18639, + 31627, + 27, + 1, + 0, + 130973, + 4766, + 390, + 275, + 11862, + 314, + 17349, + 44, + 74288, + 304, + 1135, + 5240, + 27, + 1, + 0, + 130976, + 3 + ], + "new_ids": [ + 3066, + 599, + 5509, + 39, + 518, + 4766, + 390, + 275, + 11862, + 314, + 17349, + 44, + 74288, + 304, + 1135, + 5240, + 1848, + 4358, + 275, + 6787, + 2029, + 417, + 275, + 5133 + ], + "text": "We are asked: \"What is the capital of France? Answer in one word.\" So the answer should be the single", + "dtype": "bfloat16", + "top1": 3066 +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/ref_long.json b/benchmarks/edge_harness/results/ref_long.json new file mode 100644 index 000000000..3c913c09a --- /dev/null +++ b/benchmarks/edge_harness/results/ref_long.json @@ -0,0 +1,2508 @@ +{ + "prompt": "<\uff5cstart\u2581of\u2581sentence\uff5c><|System|>\nyou are a helpful assistant.<\uff5cend\u2581of\u2581sentence\uff5c><\uff5cstart\u2581of\u2581sentence\uff5c><|User|>Here is an inventory listing.\nItem 1: the code for warehouse district 1 is 1007. Item 2: the code for warehouse district 2 is 1014. Item 3: the code for warehouse district 3 is 1021. Item 4: the code for warehouse district 4 is 1028. Item 5: the code for warehouse district 5 is 1035. Item 6: the code for warehouse district 6 is 1042. Item 7: the code for warehouse district 7 is 1049. Item 8: the code for warehouse district 8 is 1056. Item 9: the code for warehouse district 9 is 1063. Item 10: the code for warehouse district 10 is 1070. Item 11: the code for warehouse district 11 is 1077. Item 12: the code for warehouse district 12 is 1084. Item 13: the code for warehouse district 13 is 1091. Item 14: the code for warehouse district 14 is 1098. Item 15: the code for warehouse district 15 is 1105. Item 16: the code for warehouse district 16 is 1112. Item 17: the code for warehouse district 17 is 1119. Item 18: the code for warehouse district 18 is 1126. Item 19: the code for warehouse district 19 is 1133. Item 20: the code for warehouse district 20 is 1140. Item 21: the code for warehouse district 21 is 1147. Item 22: the code for warehouse district 22 is 1154. Item 23: the code for warehouse district 23 is 1161. Item 24: the code for warehouse district 24 is 1168. Item 25: the code for warehouse district 25 is 1175. Item 26: the code for warehouse district 26 is 1182. Item 27: the code for warehouse district 27 is 1189. Item 28: the code for warehouse district 28 is 1196. Item 29: the code for warehouse district 29 is 1203. Item 30: the code for warehouse district 30 is 1210. Item 31: the code for warehouse district 31 is 1217. Item 32: the code for warehouse district 32 is 1224. Item 33: the code for warehouse district 33 is 1231. Item 34: the code for warehouse district 34 is 1238. Item 35: the code for warehouse district 35 is 1245. Item 36: the code for warehouse district 36 is 1252. Item 37: the code for warehouse district 37 is 1259. Item 38: the code for warehouse district 38 is 1266. Item 39: the code for warehouse district 39 is 1273. Item 40: the code for warehouse district 40 is 1280. Item 41: the code for warehouse district 41 is 1287. Item 42: the code for warehouse district 42 is 1294. Item 43: the code for warehouse district 43 is 1301. Item 44: the code for warehouse district 44 is 1308. Item 45: the code for warehouse district 45 is 1315. Item 46: the code for warehouse district 46 is 1322. Item 47: the code for warehouse district 47 is 1329. Item 48: the code for warehouse district 48 is 1336. Item 49: the code for warehouse district 49 is 1343. Item 50: the code for warehouse district 50 is 1350. Item 51: the code for warehouse district 51 is 1357. Item 52: the code for warehouse district 52 is 1364. Item 53: the code for warehouse district 53 is 1371. Item 54: the code for warehouse district 54 is 1378. Item 55: the code for warehouse district 55 is 1385. Item 56: the code for warehouse district 56 is 1392. Item 57: the code for warehouse district 57 is 1399. Item 58: the code for warehouse district 58 is 1406. Item 59: the code for warehouse district 59 is 1413. Item 60: the code for warehouse district 60 is 1420. Item 61: the code for warehouse district 61 is 1427. Item 62: the code for warehouse district 62 is 1434. Item 63: the code for warehouse district 63 is 1441. Item 64: the code for warehouse district 64 is 1448. Item 65: the code for warehouse district 65 is 1455. Item 66: the code for warehouse district 66 is 1462. Item 67: the code for warehouse district 67 is 1469. Item 68: the code for warehouse district 68 is 1476. Item 69: the code for warehouse district 69 is 1483. Item 70: the code for warehouse district 70 is 1490. Item 71: the code for warehouse district 71 is 1497. Item 72: the code for warehouse district 72 is 1504. Item 73: the code for warehouse district 73 is 1511. Item 74: the code for warehouse district 74 is 1518. Item 75: the code for warehouse district 75 is 1525. Item 76: the code for warehouse district 76 is 1532. Item 77: the code for warehouse district 77 is 1539. Item 78: the code for warehouse district 78 is 1546. Item 79: the code for warehouse district 79 is 1553. Item 80: the code for warehouse district 80 is 1560. Item 81: the code for warehouse district 81 is 1567. Item 82: the code for warehouse district 82 is 1574. Item 83: the code for warehouse district 83 is 1581. Item 84: the code for warehouse district 84 is 1588. Item 85: the code for warehouse district 85 is 1595. Item 86: the code for warehouse district 86 is 1602. Item 87: the code for warehouse district 87 is 1609. Item 88: the code for warehouse district 88 is 1616. Item 89: the code for warehouse district 89 is 1623. Item 90: the code for warehouse district 90 is 1630. Item 91: the code for warehouse district 91 is 1637. Item 92: the code for warehouse district 92 is 1644. Item 93: the code for warehouse district 93 is 1651. Item 94: the code for warehouse district 94 is 1658. Item 95: the code for warehouse district 95 is 1665. Item 96: the code for warehouse district 96 is 1672. Item 97: the code for warehouse district 97 is 1679. Item 98: the code for warehouse district 98 is 1686. Item 99: the code for warehouse district 99 is 1693. Item 100: the code for warehouse district 100 is 1700. Item 101: the code for warehouse district 101 is 1707. Item 102: the code for warehouse district 102 is 1714. Item 103: the code for warehouse district 103 is 1721. Item 104: the code for warehouse district 104 is 1728. Item 105: the code for warehouse district 105 is 1735. Item 106: the code for warehouse district 106 is 1742. Item 107: the code for warehouse district 107 is 1749. Item 108: the code for warehouse district 108 is 1756. Item 109: the code for warehouse district 109 is 1763. Item 110: the code for warehouse district 110 is 1770. Item 111: the code for warehouse district 111 is 1777. Item 112: the code for warehouse district 112 is 1784. Item 113: the code for warehouse district 113 is 1791. Item 114: the code for warehouse district 114 is 1798. Item 115: the code for warehouse district 115 is 1805. Item 116: the code for warehouse district 116 is 1812. Item 117: the code for warehouse district 117 is 1819. Item 118: the code for warehouse district 118 is 1826. Item 119: the code for warehouse district 119 is 1833. Item 120: the code for warehouse district 120 is 1840.\nWhat is the code for warehouse district 73? Answer with just the number.<\uff5cend\u2581of\u2581sentence\uff5c><\uff5cstart\u2581of\u2581sentence\uff5c><|Bot|>", + "prompt_ids": [ + 0, + 130972, + 198, + 18178, + 599, + 259, + 18639, + 31627, + 27, + 1, + 0, + 130973, + 13023, + 390, + 300, + 31946, + 35404, + 27, + 198, + 5369, + 220, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 390, + 220, + 30, + 29, + 29, + 36, + 27, + 28017, + 220, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 390, + 220, + 30, + 29, + 30, + 33, + 27, + 28017, + 220, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 390, + 220, + 30, + 29, + 31, + 30, + 27, + 28017, + 220, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 390, + 220, + 30, + 29, + 31, + 37, + 27, + 28017, + 220, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 390, + 220, + 30, + 29, + 32, + 34, + 27, + 28017, + 220, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 390, + 220, + 30, + 29, + 33, + 31, + 27, + 28017, + 220, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 390, + 220, + 30, + 29, + 33, + 38, + 27, + 28017, + 220, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 390, + 220, + 30, + 29, + 34, + 35, + 27, + 28017, + 220, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 390, + 220, + 30, + 29, + 35, + 32, + 27, + 28017, + 220, + 30, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 390, + 220, + 30, + 29, + 36, + 29, + 27, + 28017, + 220, + 30, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 390, + 220, + 30, + 29, + 36, + 36, + 27, + 28017, + 220, + 30, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 31, + 390, + 220, + 30, + 29, + 37, + 33, + 27, + 28017, + 220, + 30, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 32, + 390, + 220, + 30, + 29, + 38, + 30, + 27, + 28017, + 220, + 30, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 33, + 390, + 220, + 30, + 29, + 38, + 37, + 27, + 28017, + 220, + 30, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 34, + 390, + 220, + 30, + 30, + 29, + 34, + 27, + 28017, + 220, + 30, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 35, + 390, + 220, + 30, + 30, + 30, + 31, + 27, + 28017, + 220, + 30, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 36, + 390, + 220, + 30, + 30, + 30, + 38, + 27, + 28017, + 220, + 30, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 37, + 390, + 220, + 30, + 30, + 31, + 35, + 27, + 28017, + 220, + 30, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 38, + 390, + 220, + 30, + 30, + 32, + 32, + 27, + 28017, + 220, + 31, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 29, + 390, + 220, + 30, + 30, + 33, + 29, + 27, + 28017, + 220, + 31, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 30, + 390, + 220, + 30, + 30, + 33, + 36, + 27, + 28017, + 220, + 31, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 31, + 390, + 220, + 30, + 30, + 34, + 33, + 27, + 28017, + 220, + 31, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 32, + 390, + 220, + 30, + 30, + 35, + 30, + 27, + 28017, + 220, + 31, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 33, + 390, + 220, + 30, + 30, + 35, + 37, + 27, + 28017, + 220, + 31, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 34, + 390, + 220, + 30, + 30, + 36, + 34, + 27, + 28017, + 220, + 31, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 35, + 390, + 220, + 30, + 30, + 37, + 31, + 27, + 28017, + 220, + 31, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 36, + 390, + 220, + 30, + 30, + 37, + 38, + 27, + 28017, + 220, + 31, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 37, + 390, + 220, + 30, + 30, + 38, + 35, + 27, + 28017, + 220, + 31, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 31, + 38, + 390, + 220, + 30, + 31, + 29, + 32, + 27, + 28017, + 220, + 32, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 29, + 390, + 220, + 30, + 31, + 30, + 29, + 27, + 28017, + 220, + 32, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 30, + 390, + 220, + 30, + 31, + 30, + 36, + 27, + 28017, + 220, + 32, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 31, + 390, + 220, + 30, + 31, + 31, + 33, + 27, + 28017, + 220, + 32, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 32, + 390, + 220, + 30, + 31, + 32, + 30, + 27, + 28017, + 220, + 32, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 33, + 390, + 220, + 30, + 31, + 32, + 37, + 27, + 28017, + 220, + 32, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 34, + 390, + 220, + 30, + 31, + 33, + 34, + 27, + 28017, + 220, + 32, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 35, + 390, + 220, + 30, + 31, + 34, + 31, + 27, + 28017, + 220, + 32, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 36, + 390, + 220, + 30, + 31, + 34, + 38, + 27, + 28017, + 220, + 32, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 37, + 390, + 220, + 30, + 31, + 35, + 35, + 27, + 28017, + 220, + 32, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 32, + 38, + 390, + 220, + 30, + 31, + 36, + 32, + 27, + 28017, + 220, + 33, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 29, + 390, + 220, + 30, + 31, + 37, + 29, + 27, + 28017, + 220, + 33, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 30, + 390, + 220, + 30, + 31, + 37, + 36, + 27, + 28017, + 220, + 33, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 31, + 390, + 220, + 30, + 31, + 38, + 33, + 27, + 28017, + 220, + 33, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 32, + 390, + 220, + 30, + 32, + 29, + 30, + 27, + 28017, + 220, + 33, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 33, + 390, + 220, + 30, + 32, + 29, + 37, + 27, + 28017, + 220, + 33, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 34, + 390, + 220, + 30, + 32, + 30, + 34, + 27, + 28017, + 220, + 33, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 35, + 390, + 220, + 30, + 32, + 31, + 31, + 27, + 28017, + 220, + 33, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 36, + 390, + 220, + 30, + 32, + 31, + 38, + 27, + 28017, + 220, + 33, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 37, + 390, + 220, + 30, + 32, + 32, + 35, + 27, + 28017, + 220, + 33, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 33, + 38, + 390, + 220, + 30, + 32, + 33, + 32, + 27, + 28017, + 220, + 34, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 29, + 390, + 220, + 30, + 32, + 34, + 29, + 27, + 28017, + 220, + 34, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 30, + 390, + 220, + 30, + 32, + 34, + 36, + 27, + 28017, + 220, + 34, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 31, + 390, + 220, + 30, + 32, + 35, + 33, + 27, + 28017, + 220, + 34, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 32, + 390, + 220, + 30, + 32, + 36, + 30, + 27, + 28017, + 220, + 34, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 33, + 390, + 220, + 30, + 32, + 36, + 37, + 27, + 28017, + 220, + 34, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 34, + 390, + 220, + 30, + 32, + 37, + 34, + 27, + 28017, + 220, + 34, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 35, + 390, + 220, + 30, + 32, + 38, + 31, + 27, + 28017, + 220, + 34, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 36, + 390, + 220, + 30, + 32, + 38, + 38, + 27, + 28017, + 220, + 34, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 37, + 390, + 220, + 30, + 33, + 29, + 35, + 27, + 28017, + 220, + 34, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 34, + 38, + 390, + 220, + 30, + 33, + 30, + 32, + 27, + 28017, + 220, + 35, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 29, + 390, + 220, + 30, + 33, + 31, + 29, + 27, + 28017, + 220, + 35, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 30, + 390, + 220, + 30, + 33, + 31, + 36, + 27, + 28017, + 220, + 35, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 31, + 390, + 220, + 30, + 33, + 32, + 33, + 27, + 28017, + 220, + 35, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 32, + 390, + 220, + 30, + 33, + 33, + 30, + 27, + 28017, + 220, + 35, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 33, + 390, + 220, + 30, + 33, + 33, + 37, + 27, + 28017, + 220, + 35, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 34, + 390, + 220, + 30, + 33, + 34, + 34, + 27, + 28017, + 220, + 35, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 35, + 390, + 220, + 30, + 33, + 35, + 31, + 27, + 28017, + 220, + 35, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 36, + 390, + 220, + 30, + 33, + 35, + 38, + 27, + 28017, + 220, + 35, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 37, + 390, + 220, + 30, + 33, + 36, + 35, + 27, + 28017, + 220, + 35, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 35, + 38, + 390, + 220, + 30, + 33, + 37, + 32, + 27, + 28017, + 220, + 36, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 29, + 390, + 220, + 30, + 33, + 38, + 29, + 27, + 28017, + 220, + 36, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 30, + 390, + 220, + 30, + 33, + 38, + 36, + 27, + 28017, + 220, + 36, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 31, + 390, + 220, + 30, + 34, + 29, + 33, + 27, + 28017, + 220, + 36, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 32, + 390, + 220, + 30, + 34, + 30, + 30, + 27, + 28017, + 220, + 36, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 33, + 390, + 220, + 30, + 34, + 30, + 37, + 27, + 28017, + 220, + 36, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 34, + 390, + 220, + 30, + 34, + 31, + 34, + 27, + 28017, + 220, + 36, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 35, + 390, + 220, + 30, + 34, + 32, + 31, + 27, + 28017, + 220, + 36, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 36, + 390, + 220, + 30, + 34, + 32, + 38, + 27, + 28017, + 220, + 36, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 37, + 390, + 220, + 30, + 34, + 33, + 35, + 27, + 28017, + 220, + 36, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 38, + 390, + 220, + 30, + 34, + 34, + 32, + 27, + 28017, + 220, + 37, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 29, + 390, + 220, + 30, + 34, + 35, + 29, + 27, + 28017, + 220, + 37, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 30, + 390, + 220, + 30, + 34, + 35, + 36, + 27, + 28017, + 220, + 37, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 31, + 390, + 220, + 30, + 34, + 36, + 33, + 27, + 28017, + 220, + 37, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 32, + 390, + 220, + 30, + 34, + 37, + 30, + 27, + 28017, + 220, + 37, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 33, + 390, + 220, + 30, + 34, + 37, + 37, + 27, + 28017, + 220, + 37, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 34, + 390, + 220, + 30, + 34, + 38, + 34, + 27, + 28017, + 220, + 37, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 35, + 390, + 220, + 30, + 35, + 29, + 31, + 27, + 28017, + 220, + 37, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 36, + 390, + 220, + 30, + 35, + 29, + 38, + 27, + 28017, + 220, + 37, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 37, + 390, + 220, + 30, + 35, + 30, + 35, + 27, + 28017, + 220, + 37, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 37, + 38, + 390, + 220, + 30, + 35, + 31, + 32, + 27, + 28017, + 220, + 38, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 29, + 390, + 220, + 30, + 35, + 32, + 29, + 27, + 28017, + 220, + 38, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 30, + 390, + 220, + 30, + 35, + 32, + 36, + 27, + 28017, + 220, + 38, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 31, + 390, + 220, + 30, + 35, + 33, + 33, + 27, + 28017, + 220, + 38, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 32, + 390, + 220, + 30, + 35, + 34, + 30, + 27, + 28017, + 220, + 38, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 33, + 390, + 220, + 30, + 35, + 34, + 37, + 27, + 28017, + 220, + 38, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 34, + 390, + 220, + 30, + 35, + 35, + 34, + 27, + 28017, + 220, + 38, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 35, + 390, + 220, + 30, + 35, + 36, + 31, + 27, + 28017, + 220, + 38, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 36, + 390, + 220, + 30, + 35, + 36, + 38, + 27, + 28017, + 220, + 38, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 37, + 390, + 220, + 30, + 35, + 37, + 35, + 27, + 28017, + 220, + 38, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 38, + 38, + 390, + 220, + 30, + 35, + 38, + 32, + 27, + 28017, + 220, + 30, + 29, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 29, + 390, + 220, + 30, + 36, + 29, + 29, + 27, + 28017, + 220, + 30, + 29, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 30, + 390, + 220, + 30, + 36, + 29, + 36, + 27, + 28017, + 220, + 30, + 29, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 31, + 390, + 220, + 30, + 36, + 30, + 33, + 27, + 28017, + 220, + 30, + 29, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 32, + 390, + 220, + 30, + 36, + 31, + 30, + 27, + 28017, + 220, + 30, + 29, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 33, + 390, + 220, + 30, + 36, + 31, + 37, + 27, + 28017, + 220, + 30, + 29, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 34, + 390, + 220, + 30, + 36, + 32, + 34, + 27, + 28017, + 220, + 30, + 29, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 35, + 390, + 220, + 30, + 36, + 33, + 31, + 27, + 28017, + 220, + 30, + 29, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 36, + 390, + 220, + 30, + 36, + 33, + 38, + 27, + 28017, + 220, + 30, + 29, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 37, + 390, + 220, + 30, + 36, + 34, + 35, + 27, + 28017, + 220, + 30, + 29, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 29, + 38, + 390, + 220, + 30, + 36, + 35, + 32, + 27, + 28017, + 220, + 30, + 30, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 29, + 390, + 220, + 30, + 36, + 36, + 29, + 27, + 28017, + 220, + 30, + 30, + 30, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 30, + 390, + 220, + 30, + 36, + 36, + 36, + 27, + 28017, + 220, + 30, + 30, + 31, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 31, + 390, + 220, + 30, + 36, + 37, + 33, + 27, + 28017, + 220, + 30, + 30, + 32, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 32, + 390, + 220, + 30, + 36, + 38, + 30, + 27, + 28017, + 220, + 30, + 30, + 33, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 33, + 390, + 220, + 30, + 36, + 38, + 37, + 27, + 28017, + 220, + 30, + 30, + 34, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 34, + 390, + 220, + 30, + 37, + 29, + 34, + 27, + 28017, + 220, + 30, + 30, + 35, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 35, + 390, + 220, + 30, + 37, + 30, + 31, + 27, + 28017, + 220, + 30, + 30, + 36, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 36, + 390, + 220, + 30, + 37, + 30, + 38, + 27, + 28017, + 220, + 30, + 30, + 37, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 37, + 390, + 220, + 30, + 37, + 31, + 35, + 27, + 28017, + 220, + 30, + 30, + 38, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 30, + 38, + 390, + 220, + 30, + 37, + 32, + 32, + 27, + 28017, + 220, + 30, + 31, + 29, + 39, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 30, + 31, + 29, + 390, + 220, + 30, + 37, + 33, + 29, + 27, + 198, + 4766, + 390, + 275, + 5760, + 396, + 56774, + 20882, + 220, + 36, + 32, + 44, + 74288, + 478, + 1839, + 275, + 2823, + 27, + 1, + 0, + 130976, + 3 + ], + "new_ids": [ + 3066, + 599, + 4114, + 300, + 31946, + 35404, + 478, + 7006, + 27149, + 396, + 56774, + 49408, + 220, + 30, + 1953, + 220, + 30, + 31, + 29, + 27, + 614, + 5565, + 27179, + 39, + 518, + 4766, + 390, + 275, + 5760, + 396, + 56774, + 20882 + ], + "text": "We are given an inventory listing with item codes for warehouse districts 1 through 120. The question asks: \"What is the code for warehouse district", + "dtype": "bfloat16", + "top1": 3066, + "scenario": "long" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/ref_tools.json b/benchmarks/edge_harness/results/ref_tools.json new file mode 100644 index 000000000..2c2eb2835 --- /dev/null +++ b/benchmarks/edge_harness/results/ref_tools.json @@ -0,0 +1,180 @@ +{ + "prompt": "<\uff5cstart\u2581of\u2581sentence\uff5c><|System|>\nyou are a helpful assistant.## Tools\nYou have access to the following functions:\n\n{\"name\": \"get_weather\", \"description\": \"Get the current weather for a city.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name\"}, \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]}}, \"required\": [\"city\"]}}\n\n\nYou are a helpful assistant with tools.<\uff5cend\u2581of\u2581sentence\uff5c><\uff5cstart\u2581of\u2581sentence\uff5c><|User|>What is the weather in Shenzhen right now, in celsius?<\uff5cend\u2581of\u2581sentence\uff5c><\uff5cstart\u2581of\u2581sentence\uff5c><|Bot|>", + "prompt_ids": [ + 0, + 130972, + 198, + 18178, + 599, + 259, + 18639, + 31627, + 27, + 2257, + 55260, + 198, + 3683, + 760, + 4561, + 312, + 275, + 4305, + 9928, + 39, + 198, + 41, + 37788, + 43, + 198, + 23421, + 2215, + 4068, + 518, + 913, + 76, + 64301, + 1078, + 518, + 21311, + 4068, + 518, + 3582, + 275, + 3345, + 14510, + 396, + 259, + 6594, + 34028, + 518, + 76248, + 4068, + 24254, + 3097, + 4068, + 518, + 9571, + 1078, + 518, + 65263, + 4068, + 24254, + 58593, + 4068, + 24254, + 3097, + 4068, + 518, + 2943, + 1078, + 518, + 21311, + 4068, + 518, + 42088, + 2795, + 33889, + 518, + 11220, + 4068, + 24254, + 3097, + 4068, + 518, + 2943, + 1078, + 518, + 40327, + 4068, + 21518, + 8281, + 1078, + 518, + 83, + 91538, + 47265, + 7872, + 35951, + 518, + 40976, + 4068, + 21518, + 58593, + 7872, + 6678, + 198, + 2717, + 37788, + 43, + 198, + 198, + 3683, + 599, + 259, + 18639, + 31627, + 478, + 10397, + 27, + 1, + 0, + 130973, + 4766, + 390, + 275, + 14510, + 304, + 75040, + 103, + 872, + 2430, + 2343, + 25, + 304, + 281, + 108364, + 44, + 1, + 0, + 130976, + 3 + ], + "new_ids": [ + 1046, + 3754, + 13025, + 312, + 1843, + 275, + 14510, + 304, + 75040, + 103, + 872, + 2430, + 2343, + 304, + 423, + 108364, + 27, + 373, + 760, + 259, + 2122, + 4234, + 4848, + 2561, + 913, + 76, + 64301, + 77, + 455, + 764, + 41002, + 3345 + ], + "text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve current", + "dtype": "bfloat16", + "top1": 1046, + "scenario": "tools" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/ref_tools32.json b/benchmarks/edge_harness/results/ref_tools32.json new file mode 100644 index 000000000..b3165f063 --- /dev/null +++ b/benchmarks/edge_harness/results/ref_tools32.json @@ -0,0 +1,180 @@ +{ + "prompt": "<\uff5cstart\u2581of\u2581sentence\uff5c><|System|>\nyou are a helpful assistant.## Tools\nYou have access to the following functions:\n\n{\"name\": \"get_weather\", \"description\": \"Get the current weather for a city.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"city\": {\"type\": \"string\", \"description\": \"City name\"}, \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]}}, \"required\": [\"city\"]}}\n\n\nYou are a helpful assistant with tools.<\uff5cend\u2581of\u2581sentence\uff5c><\uff5cstart\u2581of\u2581sentence\uff5c><|User|>What is the weather in Shenzhen right now, in celsius?<\uff5cend\u2581of\u2581sentence\uff5c><\uff5cstart\u2581of\u2581sentence\uff5c><|Bot|>", + "prompt_ids": [ + 0, + 130972, + 198, + 18178, + 599, + 259, + 18639, + 31627, + 27, + 2257, + 55260, + 198, + 3683, + 760, + 4561, + 312, + 275, + 4305, + 9928, + 39, + 198, + 41, + 37788, + 43, + 198, + 23421, + 2215, + 4068, + 518, + 913, + 76, + 64301, + 1078, + 518, + 21311, + 4068, + 518, + 3582, + 275, + 3345, + 14510, + 396, + 259, + 6594, + 34028, + 518, + 76248, + 4068, + 24254, + 3097, + 4068, + 518, + 9571, + 1078, + 518, + 65263, + 4068, + 24254, + 58593, + 4068, + 24254, + 3097, + 4068, + 518, + 2943, + 1078, + 518, + 21311, + 4068, + 518, + 42088, + 2795, + 33889, + 518, + 11220, + 4068, + 24254, + 3097, + 4068, + 518, + 2943, + 1078, + 518, + 40327, + 4068, + 21518, + 8281, + 1078, + 518, + 83, + 91538, + 47265, + 7872, + 35951, + 518, + 40976, + 4068, + 21518, + 58593, + 7872, + 6678, + 198, + 2717, + 37788, + 43, + 198, + 198, + 3683, + 599, + 259, + 18639, + 31627, + 478, + 10397, + 27, + 1, + 0, + 130973, + 4766, + 390, + 275, + 14510, + 304, + 75040, + 103, + 872, + 2430, + 2343, + 25, + 304, + 281, + 108364, + 44, + 1, + 0, + 130976, + 3 + ], + "new_ids": [ + 1046, + 3754, + 13025, + 312, + 1843, + 275, + 14510, + 304, + 75040, + 103, + 872, + 2430, + 2343, + 304, + 423, + 108364, + 27, + 373, + 760, + 259, + 2122, + 4234, + 4848, + 2561, + 913, + 76, + 64301, + 77, + 455, + 764, + 41002, + 14510 + ], + "text": "The user wants to know the weather in Shenzhen right now in Celsius. I have a function available called `get_weather` that can retrieve weather", + "dtype": "float32", + "top1": 1046, + "scenario": "tools" +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/s25_decode_estimate.json b/benchmarks/edge_harness/results/s25_decode_estimate.json new file mode 100644 index 000000000..2ddceab94 --- /dev/null +++ b/benchmarks/edge_harness/results/s25_decode_estimate.json @@ -0,0 +1,44 @@ +{ + "fp16|ctx1024": { + "hybrid_ms": 46.38, + "tok_s": 21.6, + "uniform_ms": 43.65, + "uniform_tok_s": 22.9, + "speedup": 0.94 + }, + "fp16|ctx4096": { + "hybrid_ms": 56.22, + "tok_s": 17.8, + "uniform_ms": 82.99, + "uniform_tok_s": 12.0, + "speedup": 1.48 + }, + "w8a16|ctx1024": { + "hybrid_ms": 27.05, + "tok_s": 37.0, + "uniform_ms": 31.61, + "uniform_tok_s": 31.6, + "speedup": 1.17 + }, + "w8a16|ctx4096": { + "hybrid_ms": 30.92, + "tok_s": 32.3, + "uniform_ms": 47.07, + "uniform_tok_s": 21.2, + "speedup": 1.52 + }, + "w4a16|ctx1024": { + "hybrid_ms": 27.15, + "tok_s": 36.8, + "uniform_ms": 31.56, + "uniform_tok_s": 31.7, + "speedup": 1.16 + }, + "w4a16|ctx4096": { + "hybrid_ms": 30.89, + "tok_s": 32.4, + "uniform_ms": 46.54, + "uniform_tok_s": 21.5, + "speedup": 1.51 + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/s25_step_final.json b/benchmarks/edge_harness/results/s25_step_final.json new file mode 100644 index 000000000..a78b6bf45 --- /dev/null +++ b/benchmarks/edge_harness/results/s25_step_final.json @@ -0,0 +1,23 @@ +{ + "step_ms": 27.36, + "tok_s": 36.5, + "baseline_ms": 31.01, + "baseline_tok_s": 32.2, + "components": { + "sliding x21 (ring)": { + "ms_each": 0.803, + "ms_total": 16.86, + "snr_db": 36.85 + }, + "full x7 (roll)": { + "ms_each": 0.942, + "ms_total": 6.59, + "snr_db": 37.39 + }, + "output head x1": { + "ms_each": 3.907, + "ms_total": 3.91, + "snr_db": 62.76 + } + } +} \ No newline at end of file diff --git a/benchmarks/edge_harness/results/vllm_cpu_bench.jsonl b/benchmarks/edge_harness/results/vllm_cpu_bench.jsonl new file mode 100644 index 000000000..4d06073a1 --- /dev/null +++ b/benchmarks/edge_harness/results/vllm_cpu_bench.jsonl @@ -0,0 +1,6 @@ +{"tag": "vllm-bf16-t4", "model": "XHToken/Spark-X2.5-1.7B", "dtype": "bfloat16", "quantization": null, "enforce_eager": false, "init_s": 58.71, "threads_bind": "0-3", "omp_num_threads": "4", "pp_tokens": 512, "pp_s": 0.4598, "pp_tps": 1113.44, "tg_tokens": 128, "tg_s": 11.4339, "tg_tps": 11.11, "raw": {"pp": [0.4686984680593014, 0.4646501708775759, 0.4598343148827553], "tg": [11.532050816342235, 11.648163571022451, 11.762528502848], "tg1": [0.10028818994760513, 0.09829043503850698, 0.09812222281470895]}} +{"tag": "vllm-bf16-t8", "model": "XHToken/Spark-X2.5-1.7B", "dtype": "bfloat16", "quantization": null, "enforce_eager": false, "init_s": 60.4, "threads_bind": "0-7", "omp_num_threads": "8", "pp_tokens": 512, "pp_s": 0.2422, "pp_tps": 2113.72, "tg_tokens": 128, "tg_s": 6.3884, "tg_tps": 19.88, "raw": {"pp": [0.24585375608876348, 0.24222652427852154, 0.24510639300569892], "tg": [6.66656888788566, 6.739134748000652, 6.437250141985714], "tg1": [0.048853468615561724, 0.049976578913629055, 0.050105633679777384]}} +{"tag": "vllm-bf16-t16", "model": "XHToken/Spark-X2.5-1.7B", "dtype": "bfloat16", "quantization": null, "enforce_eager": false, "init_s": 58.5, "threads_bind": "0-15", "omp_num_threads": "16", "pp_tokens": 512, "pp_s": 0.1407, "pp_tps": 3639.39, "tg_tokens": 128, "tg_s": 4.211, "tg_tps": 30.16, "raw": {"pp": [0.2433060402981937, 0.14068301068618894, 0.153479119297117], "tg": [4.422979693859816, 4.2430975288152695, 4.593760296236724], "tg1": [0.03211195208132267, 0.032789395190775394, 0.032833294942975044]}} +{"tag": "vllm-bf16-t32", "model": "XHToken/Spark-X2.5-1.7B", "dtype": "bfloat16", "quantization": null, "enforce_eager": false, "init_s": 82.82, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1149, "pp_tps": 4454.92, "tg_tokens": 128, "tg_s": 3.4939, "tg_tps": 36.35, "raw": {"pp": [0.1943472889252007, 0.17450002301484346, 0.11492921691387892], "tg": [3.647964600007981, 3.681034818291664, 3.5203079166822135], "tg1": [0.026433659251779318, 0.031900678761303425, 0.026734518352895975]}} +{"tag": "vllm-int8-t16", "model": "/data/zhoutaichang/embedding_infer/models/Spark-X2.5-1.7B-int8", "dtype": "bfloat16", "quantization": null, "enforce_eager": false, "init_s": 72.92, "threads_bind": "0-15", "omp_num_threads": "16", "pp_tokens": 512, "pp_s": 0.0997, "pp_tps": 5135.24, "tg_tokens": 128, "tg_s": 2.6293, "tg_tps": 48.3, "raw": {"pp": [0.18161257961764932, 0.10272291582077742, 0.09970314893871546], "tg": [2.6517742942087352, 2.6720670321956277, 2.650030676741153], "tg1": [0.02092872094362974, 0.020713889971375465, 0.02339001838117838]}} +{"tag": "vllm-int8-t32", "model": "/data/zhoutaichang/embedding_infer/models/Spark-X2.5-1.7B-int8", "dtype": "bfloat16", "quantization": null, "enforce_eager": false, "init_s": 72.16, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.0734, "pp_tps": 6978.07, "tg_tokens": 128, "tg_s": 2.173, "tg_tps": 58.44, "raw": {"pp": [0.09362926566973329, 0.07337276358157396, 0.07486572163179517], "tg": [2.2358193420805037, 2.203285865020007, 2.190118156839162], "tg1": [0.017937721218913794, 0.017370613757520914, 0.01709391176700592]}} diff --git a/benchmarks/edge_harness/results/vllm_cpu_gptq.jsonl b/benchmarks/edge_harness/results/vllm_cpu_gptq.jsonl new file mode 100644 index 000000000..ccb2d7983 --- /dev/null +++ b/benchmarks/edge_harness/results/vllm_cpu_gptq.jsonl @@ -0,0 +1,6 @@ +{"tag": "gptq-w4a8-g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 78.36, "rss_mb": 8450.5, "rss_peak_mb": 8476.7, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1191, "pp_tps": 4299.46, "tg_tokens": 128, "tg_s": 1.6702, "tg_tps": 76.04, "raw": {"pp": [0.11972225410863757, 0.11908483225852251, 0.12134581292048097], "tg": [1.732507862150669, 1.6979356687515974, 1.6837419890798628], "tg1": [0.01368768885731697, 0.013497897889465094, 0.013701466843485832]}} +{"tag": "gptq-w4a8-g128-head", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 78.6, "rss_mb": 8104.2, "rss_peak_mb": 9060.8, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1185, "pp_tps": 4320.16, "tg_tokens": 128, "tg_s": 1.4336, "tg_tps": 88.59, "raw": {"pp": [0.14377897279337049, 0.11851405678316951, 0.14887535758316517], "tg": [1.4482398289255798, 1.904409646987915, 2.0828713290393353], "tg1": [0.015112247318029404, 0.016326773911714554, 0.014672661200165749]}} +{"tag": "gptq-w4a16-g128-head", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 78.86, "rss_mb": 8617.9, "rss_peak_mb": 8620.6, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1099, "pp_tps": 4658.85, "tg_tokens": 128, "tg_s": 3.3844, "tg_tps": 37.53, "raw": {"pp": [0.11926615796983242, 0.115292611066252, 0.10989844426512718], "tg": [3.4093596399761736, 3.5208795364014804, 3.4581055720336735], "tg1": [0.02689351700246334, 0.025302201975136995, 0.02496027434244752]}} +{"tag": "gptq-w4a8-g128-head-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 52.79, "rss_mb": 8921.1, "rss_peak_mb": 8951.6, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1638, "pp_tps": 3126.36, "tg_tokens": 128, "tg_s": 1.5445, "tg_tps": 82.23, "raw": {"pp": [0.17734218668192625, 0.16376852383837104, 0.16509774001315236], "tg": [1.8541209190152586, 1.5617477837949991, 1.5574598712846637], "tg1": [0.013258503284305334, 0.012924029026180506, 0.014902695082128048]}} +{"tag": "gptq-w4a8-confirm1", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 50.82, "rss_mb": 8098.6, "rss_peak_mb": 9056.3, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1176, "pp_tps": 4353.18, "tg_tokens": 128, "tg_s": 1.4289, "tg_tps": 88.88, "raw": {"pp": [0.11795889306813478, 0.11761508556082845, 0.11978365201503038], "tg": [1.4589618942700326, 1.4405048349872231, 1.4417059477418661], "tg1": [0.011758381966501474, 0.011617015115916729, 0.011557874735444784]}} +{"tag": "gptq-w4a8-confirm2", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-gptq-g128-head", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 50.91, "rss_mb": 8119.8, "rss_peak_mb": 9060.8, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1175, "pp_tps": 4357.17, "tg_tokens": 128, "tg_s": 1.4345, "tg_tps": 88.54, "raw": {"pp": [0.125939070712775, 0.12151174107566476, 0.1175074027851224], "tg": [1.4461753987707198, 1.4628022378310561, 1.4459114633500576], "tg1": [0.011875604279339314, 0.011563725769519806, 0.01145239407196641]}} diff --git a/benchmarks/edge_harness/results/vllm_cpu_int4.jsonl b/benchmarks/edge_harness/results/vllm_cpu_int4.jsonl new file mode 100644 index 000000000..0d362ac41 --- /dev/null +++ b/benchmarks/edge_harness/results/vllm_cpu_int4.jsonl @@ -0,0 +1,2 @@ +{"tag": "int4-g64-t16", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 101.49, "threads_bind": "0-15", "omp_num_threads": "16", "pp_tokens": 512, "pp_s": 0.1912, "pp_tps": 2678.3, "tg_tokens": 128, "tg_s": 2.0824, "tg_tps": 60.99, "raw": {"pp": [0.22419720515608788, 0.1911660679616034, 0.2031778679229319], "tg": [2.0980894481763244, 2.4653891273774207, 2.6055239280685782], "tg1": [0.015646060928702354, 0.015766777098178864, 0.015892264898866415]}} +{"tag": "int4-g64-t32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 79.73, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.2377, "pp_tps": 2154.14, "tg_tokens": 128, "tg_s": 4.9647, "tg_tps": 25.58, "raw": {"pp": [0.3366117808036506, 0.27306000515818596, 0.23768149968236685], "tg": [5.007967857178301, 6.818124628160149, 5.222140334080905], "tg1": [0.0498577537946403, 0.04329173220321536, 0.062197581864893436]}} diff --git a/benchmarks/edge_harness/results/vllm_cpu_int4_compile.jsonl b/benchmarks/edge_harness/results/vllm_cpu_int4_compile.jsonl new file mode 100644 index 000000000..3de4ede3e --- /dev/null +++ b/benchmarks/edge_harness/results/vllm_cpu_int4_compile.jsonl @@ -0,0 +1,4 @@ +{"tag": "cmp-baseline", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 49.53, "rss_mb": 8707.3, "rss_peak_mb": 8758.9, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1408, "pp_tps": 3637.41, "tg_tokens": 128, "tg_s": 1.5554, "tg_tps": 81.65, "raw": {"pp": [0.14936214685440063, 0.17236464144662023, 0.14075957285240293], "tg": [1.5747989346273243, 1.5686334948986769, 1.5682353260926902], "tg1": [0.012955669313669205, 0.012870142236351967, 0.012807924300432205]}} +{"tag": "cmp-size1", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": "{\"compile_sizes\":[1]}", "quantization": null, "enforce_eager": false, "init_s": 80.0, "rss_mb": 8681.8, "rss_peak_mb": 8740.4, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1395, "pp_tps": 3670.22, "tg_tokens": 128, "tg_s": 1.5532, "tg_tps": 81.77, "raw": {"pp": [0.14617433119565248, 0.14269194705411792, 0.13950116094201803], "tg": [1.587453848682344, 1.5752646420150995, 1.5655201659537852], "tg1": [0.012334673199802637, 0.01273215189576149, 0.012294056359678507]}} +{"tag": "cmp-eager", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": true, "init_s": 5.09, "rss_mb": 8530.5, "rss_peak_mb": 8582.5, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.163, "pp_tps": 3140.87, "tg_tokens": 128, "tg_s": 2.1569, "tg_tps": 58.88, "raw": {"pp": [0.16301239980384707, 0.1713313781656325, 0.16896916087716818], "tg": [2.23764361301437, 2.2011317629367113, 2.1742129577323794], "tg1": [0.01755737094208598, 0.020252026617527008, 0.01728561008349061]}} +{"tag": "cmp-baseline2", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 49.1, "rss_mb": 8697.0, "rss_peak_mb": 8744.9, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1392, "pp_tps": 3678.48, "tg_tokens": 128, "tg_s": 1.5545, "tg_tps": 81.7, "raw": {"pp": [0.13918800232931972, 0.1472035078331828, 0.1469828588888049], "tg": [1.5670117503032088, 1.6028917939402163, 1.5678649940527976], "tg1": [0.01251423405483365, 0.012681994121521711, 0.012507637031376362]}} diff --git a/benchmarks/edge_harness/results/vllm_cpu_int4_decode.jsonl b/benchmarks/edge_harness/results/vllm_cpu_int4_decode.jsonl new file mode 100644 index 000000000..a35220ca6 --- /dev/null +++ b/benchmarks/edge_harness/results/vllm_cpu_int4_decode.jsonl @@ -0,0 +1,4 @@ +{"tag": "g64", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 48.72, "rss_mb": 8698.2, "rss_peak_mb": 8740.1, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1388, "pp_tps": 3688.83, "tg_tokens": 128, "tg_s": 1.5661, "tg_tps": 81.1, "raw": {"pp": [0.18187299091368914, 0.13879754999652505, 0.14083293126896024], "tg": [1.5783826517872512, 1.605138630606234, 1.6051195533946157], "tg1": [0.012359960936009884, 0.01597361685708165, 0.012323817238211632]}} +{"tag": "g64-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 48.8, "rss_mb": 8696.5, "rss_peak_mb": 8740.1, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1358, "pp_tps": 3769.01, "tg_tokens": 128, "tg_s": 1.5464, "tg_tps": 82.13, "raw": {"pp": [0.1431337110698223, 0.1448423806577921, 0.13584455801174045], "tg": [1.5589092588052154, 1.5590085829608142, 1.5854099839925766], "tg1": [0.014919216744601727, 0.012509028892964125, 0.012527205981314182]}} +{"tag": "g128", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": true, "quantization": null, "enforce_eager": false, "init_s": 79.77, "rss_mb": 8076.3, "rss_peak_mb": 8125.4, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1427, "pp_tps": 3588.51, "tg_tokens": 128, "tg_s": 1.4857, "tg_tps": 85.48, "raw": {"pp": [0.14267769688740373, 0.16987481201067567, 0.14496924076229334], "tg": [1.5096165058203042, 1.497948023956269, 1.5295235989615321], "tg1": [0.012536610942333937, 0.012263210024684668, 0.01491469331085682]}} +{"tag": "g128-nodetok", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g128", "dtype": "bfloat16", "compilation_config": null, "detokenize": false, "quantization": null, "enforce_eager": false, "init_s": 49.17, "rss_mb": 8080.9, "rss_peak_mb": 8126.0, "threads_bind": "0-31", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1356, "pp_tps": 3775.85, "tg_tokens": 128, "tg_s": 1.4776, "tg_tps": 85.95, "raw": {"pp": [0.13559845183044672, 0.1453161840327084, 0.13751713698729873], "tg": [1.4896750310435891, 1.524823788087815, 1.522187131922692], "tg1": [0.012088158167898655, 0.012088059913367033, 0.012109969276934862]}} diff --git a/benchmarks/edge_harness/results/vllm_cpu_int4_env.jsonl b/benchmarks/edge_harness/results/vllm_cpu_int4_env.jsonl new file mode 100644 index 000000000..6def6a94b --- /dev/null +++ b/benchmarks/edge_harness/results/vllm_cpu_int4_env.jsonl @@ -0,0 +1,4 @@ +{"tag": "env-baseline", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 47.24, "rss_mb": 8646.8, "rss_peak_mb": 8688.1, "threads_bind": "56-87", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1469, "pp_tps": 3486.13, "tg_tokens": 128, "tg_s": 1.5737, "tg_tps": 80.7, "raw": {"pp": [0.1549873249605298, 0.20091794384643435, 0.14686772087588906], "tg": [1.6050239242613316, 1.6213323511183262, 1.5863440539687872], "tg1": [0.012721874751150608, 0.012740002945065498, 0.012601670809090137]}} +{"tag": "env-blocktime-inf", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 49.31, "rss_mb": 8691.4, "rss_peak_mb": 8739.5, "threads_bind": "56-87", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1434, "pp_tps": 3571.62, "tg_tokens": 128, "tg_s": 1.7033, "tg_tps": 74.56, "raw": {"pp": [0.14446645369753242, 0.1433522249571979, 0.14958518370985985], "tg": [1.7157828658819199, 2.237393535207957, 2.7913308111019433], "tg1": [0.012530198786407709, 0.012565722223371267, 0.0136657003313303]}} +{"tag": "env-active-wait", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 49.3, "rss_mb": 8707.6, "rss_peak_mb": 8754.2, "threads_bind": "56-87", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1362, "pp_tps": 3759.95, "tg_tokens": 128, "tg_s": 1.5361, "tg_tps": 82.68, "raw": {"pp": [0.16278283717110753, 0.13748092064633965, 0.13617194490507245], "tg": [1.5799197740852833, 1.5849092500284314, 1.5487816957756877], "tg1": [0.01281209709122777, 0.01268826425075531, 0.012896947097033262]}} +{"tag": "env-spin-hard", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 49.87, "rss_mb": 8662.8, "rss_peak_mb": 8696.1, "threads_bind": "56-87", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1388, "pp_tps": 3688.13, "tg_tokens": 128, "tg_s": 1.5189, "tg_tps": 83.61, "raw": {"pp": [0.14120533410459757, 0.15029939077794552, 0.13882357813417912], "tg": [1.558573725167662, 1.5572678949683905, 1.531433311291039], "tg1": [0.012572703883051872, 0.012503004167228937, 0.01296002697199583]}} diff --git a/benchmarks/edge_harness/results/vllm_cpu_int4_node1.jsonl b/benchmarks/edge_harness/results/vllm_cpu_int4_node1.jsonl new file mode 100644 index 000000000..60d7bd444 --- /dev/null +++ b/benchmarks/edge_harness/results/vllm_cpu_int4_node1.jsonl @@ -0,0 +1,4 @@ +{"tag": "int4-g64-t4", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 82.91, "threads_bind": "56-59", "omp_num_threads": "4", "pp_tokens": 512, "pp_s": 0.6399, "pp_tps": 800.11, "tg_tokens": 128, "tg_s": 4.6906, "tg_tps": 27.08, "raw": {"pp": [0.6730642281472683, 0.6625334718264639, 0.6399114527739584], "tg": [4.743562410119921, 4.768178966827691, 4.727575741242617], "tg1": [0.0376861528493464, 0.03693371918052435, 0.037051460705697536]}} +{"tag": "int4-g64-t8", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 79.73, "threads_bind": "56-63", "omp_num_threads": "8", "pp_tokens": 512, "pp_s": 0.3468, "pp_tps": 1476.15, "tg_tokens": 128, "tg_s": 2.8847, "tg_tps": 44.02, "raw": {"pp": [0.3509289128705859, 0.3550647315569222, 0.3468484482727945], "tg": [2.913811977021396, 2.907944421749562, 2.9344828412868083], "tg1": [0.023216695990413427, 0.024607220198959112, 0.023829770740121603]}} +{"tag": "int4-g64-t16", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 79.94, "threads_bind": "56-71", "omp_num_threads": "16", "pp_tokens": 512, "pp_s": 0.197, "pp_tps": 2599.05, "tg_tokens": 128, "tg_s": 1.9036, "tg_tps": 66.72, "raw": {"pp": [0.19884239370003343, 0.19699497427791357, 0.1986131533049047], "tg": [1.918830456212163, 1.9596990118734539, 1.9682741928845644], "tg1": [0.0156706259585917, 0.015399128664284945, 0.015269591007381678]}} +{"tag": "int4-g64-t32", "model": "/data/zhoutaichang/embedding_infer/models/spark2_5-int4-g64", "dtype": "bfloat16", "compilation_config": null, "quantization": null, "enforce_eager": false, "init_s": 78.71, "threads_bind": "56-87", "omp_num_threads": "32", "pp_tokens": 512, "pp_s": 0.1453, "pp_tps": 3524.88, "tg_tokens": 128, "tg_s": 1.5834, "tg_tps": 80.2, "raw": {"pp": [0.14525310974568129, 0.14718412281945348, 0.1640946539118886], "tg": [1.6373932622373104, 1.5962264849804342, 1.6084455288946629], "tg1": [0.012919567991048098, 0.013183748349547386, 0.012779706623405218]}} diff --git a/benchmarks/edge_harness/results/w4a8_vs_tinygemm.json b/benchmarks/edge_harness/results/w4a8_vs_tinygemm.json new file mode 100644 index 000000000..cc883468c --- /dev/null +++ b/benchmarks/edge_harness/results/w4a8_vs_tinygemm.json @@ -0,0 +1,122 @@ +[ + { + "name": "qkv", + "N": 3072, + "K": 2048, + "group": 128, + "m1": { + "tinygemm_ms": 0.0229, + "w4a8_ms": 0.015, + "speedup": 1.53, + "rel_diff_vs_tinygemm": 0.0958 + }, + "m8": { + "tinygemm_ms": 0.0553, + "w4a8_ms": 0.0305, + "speedup": 1.81, + "rel_diff_vs_tinygemm": 0.1096 + }, + "m512": { + "tinygemm_ms": 3.239, + "w4a8_ms": 0.39, + "speedup": 8.3, + "rel_diff_vs_tinygemm": 0.1005 + } + }, + { + "name": "o", + "N": 2048, + "K": 2048, + "group": 128, + "m1": { + "tinygemm_ms": 0.0158, + "w4a8_ms": 0.0142, + "speedup": 1.11, + "rel_diff_vs_tinygemm": 0.1087 + }, + "m8": { + "tinygemm_ms": 0.039, + "w4a8_ms": 0.0251, + "speedup": 1.56, + "rel_diff_vs_tinygemm": 0.1127 + }, + "m512": { + "tinygemm_ms": 2.1629, + "w4a8_ms": 0.2706, + "speedup": 7.99, + "rel_diff_vs_tinygemm": 0.1011 + } + }, + { + "name": "gate_up", + "N": 13312, + "K": 2048, + "group": 128, + "m1": { + "tinygemm_ms": 0.0648, + "w4a8_ms": 0.0271, + "speedup": 2.4, + "rel_diff_vs_tinygemm": 0.0846 + }, + "m8": { + "tinygemm_ms": 0.2229, + "w4a8_ms": 0.0932, + "speedup": 2.39, + "rel_diff_vs_tinygemm": 0.0905 + }, + "m512": { + "tinygemm_ms": 13.9922, + "w4a8_ms": 1.5752, + "speedup": 8.88, + "rel_diff_vs_tinygemm": 0.1004 + } + }, + { + "name": "down", + "N": 2048, + "K": 6656, + "group": 128, + "m1": { + "tinygemm_ms": 0.0347, + "w4a8_ms": 0.0228, + "speedup": 1.52, + "rel_diff_vs_tinygemm": 0.0925 + }, + "m8": { + "tinygemm_ms": 0.1134, + "w4a8_ms": 0.056, + "speedup": 2.02, + "rel_diff_vs_tinygemm": 0.0968 + }, + "m512": { + "tinygemm_ms": 7.0191, + "w4a8_ms": 0.8952, + "speedup": 7.84, + "rel_diff_vs_tinygemm": 0.0999 + } + }, + { + "name": "lm_head", + "N": 131072, + "K": 2048, + "group": 128, + "m1": { + "tinygemm_ms": 0.5832, + "w4a8_ms": 0.4462, + "speedup": 1.31, + "rel_diff_vs_tinygemm": 0.1167 + }, + "m8": { + "tinygemm_ms": 2.1946, + "w4a8_ms": 1.023, + "speedup": 2.15, + "rel_diff_vs_tinygemm": 0.0926 + }, + "m512": { + "tinygemm_ms": 138.7226, + "w4a8_ms": 16.4399, + "speedup": 8.44, + "rel_diff_vs_tinygemm": 0.1004 + } + } +] \ No newline at end of file diff --git a/docs/analysis/README.md b/docs/analysis/README.md new file mode 100644 index 000000000..fe2f726a5 --- /dev/null +++ b/docs/analysis/README.md @@ -0,0 +1,57 @@ +# Six-engine analysis: extending vLLM-Omni to edge TTS and VLA + +Source-grounded study of six inference stacks and a practical plan for running vLLM-Omni-style TTS and vision-language-action (VLA) pipelines on edge devices. Produced 2026-09-08 from the local checkouts listed in [repository_inventory.md](repository_inventory.md); every engine report cites files and symbols at those exact commits, and every claim carries an evidence label: + +- **[Verified]** read directly in source or repo docs at the analyzed commit +- **[Inferred]** reasoned from structure, not directly confirmed +- **[Proposal]** a design suggestion made here +- **[Unknown]** could not be determined from the tree +- **repo-reported** a number published by the repository, not reproduced here +- **measured** a number produced in this session on this host (see [experiments.md](experiments.md)) + +## Reading order + +1. [repository_inventory.md](repository_inventory.md) — paths, branches, commits, working-tree state, discrepancies. +2. Engine reports (one per engine, same 13-section structure: device, kernels, model runner, scheduler, I/O, flows, tests, TTS, VLA, edge profile, limitations, references): + - [engines/vllm.md](engines/vllm.md) — the serving engine vLLM-Omni builds on + - [engines/vllm_omni.md](engines/vllm_omni.md) — multi-stage omni pipeline; what it inherits from and adds to vLLM + - [engines/vllm_omni_models.md](engines/vllm_omni_models.md) — model-level deep dive of the TTS and VLA/world-model families in vLLM-Omni + - [engines/llama_cpp.md](engines/llama_cpp.md) + - [engines/ncnn.md](engines/ncnn.md) + - [engines/mnn.md](engines/mnn.md) + - [engines/executorch.md](engines/executorch.md) +3. [comparison.md](comparison.md) — all six across device, kernels, model runner, scheduler, I/O, deployment complexity, conversion, extensibility; serving stacks vs execution runtimes. +4. [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) — targets (embedded NVIDIA, ARM CPU, Apple Silicon, mobile GPU/NPU), five strategies (direct port, backend substitution, lightweight orchestration, hybrid edge/cloud, edge-native baseline), per-runtime stage placement. +5. [tts_edge_design.md](tts_edge_design.md) and [vla_edge_design.md](vla_edge_design.md) — separate designs with integration points, adapter interfaces, state/buffer ownership, blockers. +6. [implementation_roadmap.md](implementation_roadmap.md) — phases, acceptance criteria, reproducible benchmarks. +7. [experiments.md](experiments.md) — what was actually run (GPU via the shared scheduler, CPU-only llama.cpp), commands, job records, results, cleanup, and what was deferred. +8. [edge_branch_plan.md](edge_branch_plan.md) — work packages for an `edge/omni-edge` branch inside vLLM-Omni itself (deploy profiles, cold start, CPU platform, Jetson, codec-token streaming, slimming, device-direct connector). +9. [edge_engine_comparison.md](edge_engine_comparison.md) — the branch's vLLM-Omni measured against llama.cpp (native Qwen3-TTS, CPU and CUDA) on the same host, prompts and reference speaker; why MNN/ExecuTorch numbers are still missing. +10. [edge_device_simulation.md](edge_device_simulation.md) — what edge devices can be emulated on this H200-class host (MPS SM caps, memory budgets, core pinning) and what cannot (bandwidth, clocks); measured SM-cap sensitivity for both engines, a Jetson projection, and the per-frame attribution that explains the CPU numbers (15 sequential code-predictor passes at ≈6 ms each). +11. [mobile_htp_simulation.md](mobile_htp_simulation.md) — 2026-09-10: Qualcomm AI Runtime SDK (Hexagon HTP simulator) installed without root, the branch's Code2Wav decoder converted (fp16 DLC) and run on the x86 HTP functional simulator with bit-exact CPU-backend parity; the CPU findings it triggered (KV-cached predictor, model-side sampler, fp32 compiled predictor; int8 rejected by in-situ parity) that take the CPU platform from RTF 1.7–1.8 to 0.89–1.27 on 16–32 threads (real time at 32), i.e. 1.6–2.2× faster end-to-end than llama.cpp on the same host, level at 8 threads; the code predictor also runs on the HTP simulator (fp16, 50.7 dB vs fp32); real Galaxy S24/S25 timings via Qualcomm AI Hub (§5): fp16 is faithful on device, and the right placement is heterogeneous — talker+predictor on the Hexagon NPU (49.7 ms/frame, fp16) and Code2Wav on the Adreno GPU (45.8 ms/frame at fp32, numerically exact, vs 182 ms on the NPU) — pipelining to RTF 0.62, i.e. real time on a phone with no quantization and no precision loss; every automatic int8 build is faster but numerically broken. +12. **Implementation (2026-09-09):** [edge_branch_plan.md §5](edge_branch_plan.md#5-implementation-status-branch-edgeomni-edge-2026-09-09) records what was built on the branch and the measured outcome per work package; [experiments.md §8](experiments.md#8-edge-branch-experiments-2026-09-09) holds the runs; [experiments/edge_branch/results_summary.md](experiments/edge_branch/results_summary.md) the tables. +12. [minicpmo_edge_profile.md](minicpmo_edge_profile.md) — 2026-09-11: MiniCPM-o 4.5 (3-stage omni) profiled with the same instruments: H200 batch RTF 0.748, CPU TTS head 11.6 ms/token and vocoder 833 ms/s of audio (91 % of it in the 10-step flow DiT), Galaxy S25 NPU 7.08 ms/token at 48.9 dB — 177 ms of NPU time per second of speech against Qwen3-TTS's 621 ms, because MiniCPM uses one codebook at 25 Hz instead of sixteen at 12.5 Hz; int8 breaks its code head too. +13. [spark_edge_support.md](spark_edge_support.md) — 2026-09-11: Spark-X2.5-1.7B (agentic on-device model, hybrid 3:1 sliding/full attention, head-wise attention gate, per-layer-type partial RoPE) implemented for vLLM-Omni — vLLM 0.28 had no support, and MNN/ExecuTorch/ncnn still have none. Verified exact against the official code on transformers 4.57.1 (fp32, 32/32 greedy, 2.2e-05 logprob). On CPU it beats llama.cpp at matched bf16 by 8.7–15.4× on prefill and 1.19–2.65× on decode; an int8 path adds 1.6× but costs long-context fidelity; llama.cpp Q4_K_M still wins single-stream decode (no dense int4 kernel exists in vLLM's CPU build) while vLLM int8 wins from batch 16 and leads 1.72× at batch 64. GPU vs SGLang is a tie within 4 %. On a real Galaxy S25 a full decode step (output head included, every component verified at 36.9–62.8 dB) is 27.4 ms at 1 k context in calibrated w8a16 — 36.5 tok/s, 11.8% faster than the first working version and the first build whose output is correct at all: reading the per-op device profile showed a fifth of every sliding layer went on handing a recomputed KV window back across the graph boundary, and an on-device parity check showed the uncalibrated w4a16 builds everyone quotes score −1.6 dB, i.e. uncorrelated with the reference. Calibrated w8a16 is the deployable scheme (int8 activations still break it, the third model family here to show that). The hybrid attention beats an all-full-attention model by 1.51× at 4 k but 0.94× at 1 k, where the sliding layers' full-width RoPE outweighs the shorter cache. QNN/TFLite/ORT land within 2 % of each other on the NPU (the export layout sets on-device speed, not the runtime) and the NPU beats the Adreno GPU 2.4x here, the opposite of the TTS vocoder. Android control works with vLLM's stock glm45 parsers; thinking costs 14.6× latency for ~2/10 decisions. + +## Headline findings + +- **All six checkouts match the requested branches and commits and are clean**; ncnn and ExecuTorch submodules are uninitialized, and the machine's installed vLLM-Omni is a different checkout than the analyzed one ([repository_inventory.md](repository_inventory.md)). +- **vLLM-Omni is a serving stack, not a runtime.** It subclasses vLLM's engine process, scheduler, and model runners, patches about ten vLLM symbols at import, runs every AR/generation stage as a vLLM engine process (diffusion stages run as a separate worker or inline via `inline_diffusion`), and joins stages by connectors whose default, `SharedMemoryConnector`, stages tensors through host memory (the Mooncake/Mori transfer-engine connectors have tensor fast paths and device pools). It has no CPU platform and no Apple/Android/Jetson code. What is portable is its stage semantics (1-frame first chunk, 25-frame steady chunks, 72-frame decoder context for Qwen3-TTS), its abort rules, its API shapes, and the plain-PyTorch codecs and action heads ([engines/vllm_omni.md](engines/vllm_omni.md), [engines/vllm_omni_models.md](engines/vllm_omni_models.md)). +- **Measured on this host**: Qwen3-TTS 1.7B on one L20X gives a median first-audio latency of 27.7 ms, 160 ms per 2 s chunk, RTF 0.08–0.10, but 157–188 s of observed engine initialization in the default configuration (CUDA graphs + `torch.compile`; a lower bound with `enforce_eager` or parallel stage init was not measured); llama.cpp on x86 CPU decodes a 409 M-parameter Q4_0 LLM at 135.6 tok/s on 4 threads and spends 0.6–1.2 s per 512 px SigLIP tile on 8 threads ([experiments.md](experiments.md)). +- **Qwen3-TTS already runs natively in llama.cpp and MNN**, and ExecuTorch has a streaming TTS runner (Voxtral) to copy from; so edge TTS is mostly an orchestration and parity problem rather than a model-porting problem, subject to a checkpoint/task parity gate (the server benchmark used CustomVoice 1.7B; llama.cpp's README demonstrates 1.7B Base and MNN's demo requires reference-audio cloning) ([comparison.md](comparison.md) §10). +- **No runtime has a VLA policy in tree.** ExecuTorch (static memory, zero-copy I/O, in-graph loops, CoreML/QNN delegates) and MNN (DiT runtime, vision exporters, Android NPUs) are the viable hosts; llama.cpp lacks an action head; ncnn is a component runtime only ([vla_edge_design.md](vla_edge_design.md)). +- **Recommended strategy**: keep vLLM-Omni as the reference/edge-server tier (Jetson-class only), ship a hybrid TTS first (server talker, on-device decoder over a codec-token stream of about 400 B/s raw payload) if the new wire contract and the on-device decoder measurement pass, otherwise start with a C++ orchestrator; in either case build a C++ orchestrator on llama.cpp/MNN/ExecuTorch that mirrors vLLM-Omni's pipeline semantics; VLA goes through ExecuTorch/MNN with simulation-first validation and a control/safety layer independent of inference ([vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md), [implementation_roadmap.md](implementation_roadmap.md)). +- **Built and measured on 2026-09-09 (branch `edge/omni-edge`)**: an edge deploy profile with a ramped chunk schedule removes the ≈80 ms first-chunk stall (playback-start = TTFA = 41–44 ms, RTF unchanged); concurrent stage init cuts warm start from 152 s to 95 s (cold caches 355 s); a CPU platform runs Qwen3-TTS 0.6B end to end (RTF 1.4–2.8 on 8–32 Xeon threads, the talker step alone is 1.7–3× the frame budget); hardware auto-adaptation, codec-token streaming with an on-device decoder, import slimming, a CUDA-IPC connector and per-step overhead attribution (scheduler + transport < 2 % of the frame; no scheduler rewrite or native engine rewrite warranted) are in the tree ([edge_branch_plan.md §5](edge_branch_plan.md#5-implementation-status-branch-edgeomni-edge-2026-09-09)). +- **Engine comparison (2026-09-09)**: on one L20X the branch generates a codec frame in 6 ms vs 17–18 ms for llama.cpp's native Qwen3-TTS (RTF 0.10–0.20 vs 0.28–0.31) and streams first audio in 41–54 ms without reference audio, but reference-audio (x-vector) requests cost it 0.4–0.6 s per request and it needs ≈90–150 s of startup and ≈37 GB of reserved GPU memory against llama.cpp's 3–6 s and a few GB; on CPU both engines land at RTF ≈2 for the 0.6B model, llama.cpp with half the RAM ([edge_engine_comparison.md](edge_engine_comparison.md)). +- **Edge simulation and CPU analysis (2026-09-09)**: the cards are H200-class (804 TFLOPS bf16 measured; the driver labels them L20X). Capping SMs to 6 % with MPS costs vLLM-Omni's talker only +55 % per frame (latency-bound, CUDA graphs) while llama.cpp loses 2.8×; forcing eager mode for an 8 GB budget hurts far more (TTFA 160 ms, RTF 0.5). On CPU, 73 % of every frame is the 15 sequential code-predictor passes (6.4 ms each vs a 1.6 ms bandwidth floor) in vLLM-Omni and ≈85 % in llama.cpp; the fix is fusion/quantization/cheaper sampling for ≈2× and a lighter predictor for real time ([edge_device_simulation.md](edge_device_simulation.md)). +- **Blockers**: no edge hardware reachable from this host; VLA weights not cached (download needs permission); ExecuTorch/MNN toolchains not installed (permission); vLLM-Omni's process model and host-staged default transport are architectural for the direct-port path (cold-start minimum unmeasured). + +## Conventions + +- Links are relative; source links go to `..//path` and were checked for existence by a link-check pass (see the end of [experiments.md](experiments.md)). +- Diagrams are Mermaid. +- Mobile/ARM/Apple numbers are never measured here (x86 host); they are either repo-reported with a citation or marked Unknown. + +## Second-model review + +On 2026-09-08 the finished study was independently reviewed by a second model (`gpt-6-astra` via the Codex hook, read-only against the same checkouts; its sandbox could not start because user namespaces are disabled on this host, so it ran unsandboxed under strict read-only instructions and the checkouts were verified clean afterwards). Initial verdict: agree with corrections (9 disagreements, 4 omissions). All were verified against source and applied; the main changes were: the default shared-memory transport is host-staged but the Mooncake/Mori transfer-engine connectors have device fast paths (Mori including intra-node XGMI), unvalidated for edge; only AR/generation stages are vLLM engine processes; the 1→25 chunk schedule leaves an ≈80 ms playback gap and the design now mirrors the YAML ramp/adaptive schedule; the CPU LLM benchmark is backbone-only (code predictor and decoder excluded); the 157–188 s figure is observed initialization, not a lower bound; prompt templates are engine code (`Qwen3TTSPromptEmbedsBuilder`); MNN's Qwen3-TTS path is single-callback; π0 uses SigLIP-So400m and draws noise inside `sample_actions`; simulation clients exist for DreamZero; hybrid-first is conditional on a codec-token wire-contract prototype and an on-device decoder measurement; a checkpoint/task parity gate was added. Final verdict after three rounds: **AGREE** ("agreement covers the source-grounded conclusions and conditional implementation plan; edge performance remains subject to the documented validation gates"). diff --git a/docs/analysis/edge_branch_plan.md b/docs/analysis/edge_branch_plan.md new file mode 100644 index 000000000..ac3d0bee8 --- /dev/null +++ b/docs/analysis/edge_branch_plan.md @@ -0,0 +1,157 @@ +# Plan: an edge branch for vLLM-Omni + +Target: a development branch in [`../vllm-omni`](../vllm-omni) (main @ `7be014bc`) that makes vLLM-Omni itself run, and perform acceptably, on edge devices. This is the "S1/S2/S4 inside vLLM-Omni" part of [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md); the C++ orchestrator work on llama.cpp/MNN/ExecuTorch (S3) stays outside this branch and is covered by [implementation_roadmap.md](implementation_roadmap.md). Every touch point below was located in the checkout; nothing has been changed yet. + +## 0. Ground rules + +- Branch: `edge/omni-edge` off `main@7be014bc`, repo-local identity `tzhouam `, credential helper via `gh` (per `/data/zhoutaichang/CLAUDE.md`). Work packages land as separate sub-branches (`edge/wp1-deploy-profiles`, ...) so each can become an upstream PR on its own. +- Environment: a fresh `uv venv` for this checkout (`uv pip install -e .` per the repo's `AGENTS.md`/docs), because the machine's Python imports `/home/zhoutaichang/feature/vllm-omni-main` (`be335a86f`), not this tree. This is a dependency installation and needs your go-ahead. +- Upstream policy: vLLM-Omni's `.claude/skills` (`precheck-pr`, `review-pr`, `vllm-omni-test`) define PR shape and test levels (`core_model` L1/L2, `advanced_model` L3, `full_model` L4; `cpu` platform marker exists in `pyproject.toml`). Each WP below names the tests it adds at those levels. +- GPU runs go through `gpu run --timeout --note`; model downloads and installs are permission-gated. + +## 1. Work packages + +```mermaid +graph LR + WP0[WP0 branch + env + baseline harness] --> WP1[WP1 edge deploy profiles] + WP0 --> WP2[WP2 cold-start reduction] + WP0 --> WP3[WP3 CPU platform] + WP1 --> WP4[WP4 Jetson / aarch64 CUDA] + WP2 --> WP4 + WP1 --> WP5[WP5 codec-token streaming mode] + WP3 --> WP6[WP6 dependency slimming] + WP1 --> WP7[WP7 device-direct connector] + WP5 --> WP8[WP8 benchmarks + CI lanes] + WP3 --> WP8 + WP4 --> WP8 +``` + +### WP0 — Branch, environment, baseline harness (2–3 days) + +- Create the branch and the uv environment; confirm `python -c "import vllm_omni"` resolves to this checkout. +- Promote the session driver [experiments/vllm_omni_tts/tts_stream_bench.py](experiments/vllm_omni_tts/tts_stream_bench.py) into `benchmarks/tts/stream_latency_bench.py` (reuses the example's `_build_inputs`; records TTFA, per-chunk arrivals, uninterrupted-playback latency, RTF streamed+tail, init time, RSS, `nvidia-smi` peak). JSON schema = the one in [experiments/vllm_omni_tts/bench_results.json](experiments/vllm_omni_tts/bench_results.json). +- Re-run the Qwen3-TTS 1.7B baseline on this checkout (not the installed one) and add the 0.6B variant (driver flag to override the hard-coded model ids in `_build_inputs`). +- Acceptance: baseline JSON for 0.6B and 1.7B on L20X committed under `benchmarks/tts/results/`; numbers within noise of [experiments.md](experiments.md) §2 for 1.7B. + +### WP1 — Edge deploy profiles (3–5 days) + +Touch points: `vllm_omni/deploy/` (90 YAMLs; platform overlays under `platforms:` exist in 23 of them), `vllm_omni/config/stage_config.py` (`enforce_eager` L408, `compilation_config` L418, `gpu_memory_utilization` L402, `devices` L386, `async_chunk` L548, `inline_diffusion` L273), `vllm_omni/model_executor/stage_input_processors/chunk_size_utils.py` (`parse_chunk_ramp` L42, `ramp_chunk_size` L79). + +- Add `vllm_omni/deploy/edge/qwen3_tts_edge.yaml` (and one per TTS family that passes Phase 0 parity): both stages on one device, `gpu_memory_utilization` sized from measured need (not 0.3), `codec_chunk_ramp: [4, 4, 8, 16, 25]` on by default (removes the ≈80 ms first-gap measured in [experiments.md](experiments.md)), `decode_batch_max_size: 1`, `max_num_seqs` small, `enforce_eager` documented as a switch per stage, sampling defaults unchanged. +- Add a `profile: edge` selector so `--deploy-config` is not needed (a `DeployConfig` lookup by `(model, profile)` in `config/config_factory.py`). +- Tests: L1 `tests/config/test_edge_profiles.py` (YAML resolves, schedule parses, ramp ends at `codec_chunk_frames`); L3 e2e TTFA/playback-gap regression on CUDA using WP0's harness. +- Acceptance: uninterrupted-playback latency ≤ TTFA + 60 ms on L20X with the ramp (versus ≈190 ms today); no RTF regression > 10 %. + +### WP2 — Cold-start reduction (1–2 weeks) + +Touch points: `vllm_omni/engine/stage_runtime.py` (parallel replica init exists near L460; sequential today for co-located stages), `vllm_omni/engine/async_omni_engine.py` (`_wait_for_orchestrator_init`, 300 s default that failed run 1), `vllm_omni/worker/gpu_ar_model_runner.py` (talker-MTP graph capture, L546 in the installed log), `model_executor/models/qwen3_tts/{segmented_graph_wrapper.py,cuda_graph_decoder_wrapper.py}`, `common/qwen3_code_predictor.py` (warm-up buckets + `torch.compile`). + +- Instrument init with a timeline (process spawn, weight load, profile run, KV alloc, graph capture, compile, predictor warm-up) emitted as JSON; make it a `benchmarks/tts/init_profile.py`. +- Measure the four configurations: default; `enforce_eager` both stages; compile cache warm; parallel stage init. Keep whichever combination meets the gate and make it the edge profile default. +- Persist and reuse compile/graph artifacts (`VLLM_CACHE_ROOT` already caches AOT compile; add the code-predictor warm-up buckets and code2wav graph shapes to a versioned cache keyed by checkpoint hash + deploy profile). +- Raise the default orchestrator timeout when the edge profile is active, or scale it from the measured timeline. +- Acceptance: ready-to-serve ≤ 30 s on L20X for Qwen3-TTS 0.6B with warm caches, ≤ 90 s cold; init timeline test at L3. + +### WP3 — CPU platform (3–5 weeks) + +Touch points: `vllm_omni/platforms/__init__.py` (`builtin_omni_platform_plugins` L123: cuda/rocm/npu/xpu/musa), `vllm_omni/platforms/interface.py` (`OmniPlatform` surface: `get_omni_ar_worker_cls` L61, `get_omni_generation_worker_cls` L65, `get_diffusion_worker_cls` L175, `get_device_count` L266, memory/synchronize/autocast/`set_device_control_env_var` L329; `UnspecifiedOmniPlatform` L385), `vllm_omni/platforms/cuda/platform.py` (432 lines, the smallest template), `vllm_omni/worker/{base.py,gpu_ar_worker.py,gpu_ar_model_runner.py,gpu_generation_worker.py,gpu_generation_model_runner.py,mixins.py}`, `requirements/cpu.txt` (today: `common.txt` + `onnxruntime` only), `setup.py` (already accepts device `cpu`). + +Design: vLLM's `CPUWorker(Worker)` and `CPUModelRunner(GPUModelRunner)` (`vllm/v1/worker/cpu_worker.py`, `cpu_model_runner.py`) subclass the same GPU classes that vLLM-Omni's `OmniGPUWorkerBase`/`GPUARModelRunner` extend, so the CPU platform is built by composition rather than a rewrite: + +- `vllm_omni/platforms/cpu/platform.py`: `CPUOmniPlatform(OmniPlatform)` returning `vllm_omni.worker.cpu_ar_worker.CPUARWorker`, `cpu_generation_worker.CPUGenerationWorker`, a CPU diffusion worker (initially "unsupported" with a clear error), `get_device_count() = 1`, `device_control_env_var` = none, memory functions from `psutil`, `supports_talker_mtp_graph_capture() = False`, `supports_torch_inductor()` = vLLM CPU default, autocast = bf16/fp16 per vLLM CPU dtype rules. +- `vllm_omni/worker/cpu_ar_worker.py`: `CPUARWorker(CPUWorker, OmniWorkerMixin)` with the omni hooks from `gpu_ar_worker.py` factored into `mixins.py` where they are not already; `cpu_ar_model_runner.py`: `CPUARModelRunner(CPUModelRunner)` + the omni runner mixin (`omni_connector_model_runner_mixin.py`); same pair for the generation stage (code2wav on CPU: no CUDA graphs, eager decoder, fp32). +- Register `"cpu"` in `builtin_omni_platform_plugins` behind `VLLM_TARGET_DEVICE=cpu` or vLLM's CPU platform detection; keep `UnspecifiedOmniPlatform` for the no-device case. +- `requirements/cpu.txt`: real runtime deps (soundfile, torchaudio CPU, onnxruntime) and the vLLM CPU wheel line; docs page `docs/getting_started/installation/cpu.md`. +- Model-level: the Qwen3-TTS decoder's Triton Snake/GroupNorm kernels have torch fallbacks (verified in the model report); add a CPU branch in `CustomOp.dispatch_forward` users where `forward_native` is missing; CosyVoice3's TRT default (`COSYVOICE3_TRT`) must be off on CPU. +- Tests: L1 `tests/platforms/test_cpu_platform.py` (registration, worker classes, device count) and per-family CPU unit tests already exist under `tests/model_executor/models//`; L3 e2e Qwen3-TTS 0.6B on CPU (`pytest -m "core_model and cpu"` lane). +- Validation order: x86 on this host first (the CPU-only path needs no GPU reservation), then aarch64 Linux (vLLM ships aarch64 CPU wheels; KleidiAI W4A8 for the talker). +- Acceptance: Qwen3-TTS 0.6B end-to-end on 8 x86 threads with RTF ≤ 1.0 and TTFA ≤ 500 ms; same test green on one aarch64 box; RSS ≤ 4 GB. + +### WP4 — Jetson / aarch64 CUDA (2–3 weeks, hardware-gated) + +Touch points: none in vLLM-Omni beyond the edge profile and docs; the work is environment: vLLM aarch64+CUDA build or wheel (vLLM detects Tegra via `/etc/nv_tegra_release`; CUDA arch 8.7/11.0 are in its `CMakeLists.txt`), torch for JetPack, `recipes/edge/jetson.md`. + +- Run WP0's harness and WP2's init profile on Orin; capture memory with unified-memory caveats (`gpu_memory_utilization` semantics differ when the GPU shares RAM with the CPU: add a `max_memory_gb` absolute option to the edge profile in `stage_config.py`). +- Acceptance: Qwen3-TTS 0.6B RTF ≤ 0.5 and ready-to-serve ≤ 90 s on Orin; documented power mode; blocker if no board is available. + +### WP5 — Codec-token streaming mode (S4 wire contract) (2–3 weeks) + +Touch points: `vllm_omni/entrypoints/openai/serving_speech_stream.py` (`OmniStreamingSpeechHandler`, `_generate_and_send` L260 consumes PCM chunks), `vllm_omni/entrypoints/openai/tts_adapters/{base.py,capabilities.py}` (`register_tts_adapter`), `vllm_omni/model_executor/models/output_templates.py`, `vllm_omni/data_entry_keys.py` (`codes` key), `vllm_omni/deploy/edge/qwen3_tts_talker_only.yaml` (stage 0 only, `final_output_type` codes). + +- Server: a `mode: "codec_tokens"` session option on `/v1/audio/speech/stream` that runs the talker-only profile and emits frames `{seq, n_frames, codes: int16[n_frames×16], eos}` with explicit ordering, EOS, and cancel semantics; abort path unchanged (`AsyncOmni.abort`). +- Adapter capability flag `supports_codec_stream` per family (Qwen3-TTS first; MOSS next since its codec is transformer-only). +- Client reference: `examples/online_serving/text_to_speech/edge_decoder_client.py` decoding on device with the exported `Qwen3TTSTokenizerV2Decoder` (plain torch CPU first; ExecuTorch `.pte` second), carrying the 72-frame context and `reset()` on cancel. +- Tests: L1 protocol tests (ordering, EOS, cancel mid-chunk); L3 e2e PCM parity between server-decoded and client-decoded audio (SNR ≥ 40 dB). +- Acceptance: client-observed TTFA ≤ server TTFA + RTT + first-chunk decode; bandwidth ≈ 400 B/s payload + framing; interruption stops audio within one chunk. + +### WP6 — Dependency and import slimming (1–2 weeks) + +Touch points: `pyproject.toml` optional-dependencies, `requirements/common.txt` (diffusers pin, x-transformers, openai-whisper, onnxruntime), `vllm_omni/__init__.py` and `patch.py` import graph, `vllm_omni/diffusion/` imports pulled in by AR-only paths. + +- Add an `edge` extra that excludes diffusion/video dependencies; make diffusion, forced-aligner, and VAD imports lazy on the TTS path; measure `import vllm_omni` time and RSS before/after. +- Acceptance: `python -c "import vllm_omni"` ≤ 3 s and no `diffusers` import for a TTS-only deploy; test at L1 with an import-guard test. + +### WP7 — Device-direct connector for co-located stages (2 weeks, optional) + +Touch points: `vllm_omni/distributed/omni_connectors/connectors/{base.py,shm_connector.py}` (`OmniConnectorBase.put/get`), `utils/serialization.py` (`OmniMsgpackEncoder`, zero-copy TODO), deploy YAML `connectors:` section. + +- `CudaIpcConnector`: `put` exports a CUDA IPC handle (`torch.multiprocessing` reductions) for same-GPU stages instead of D2H + msgpack; `get` maps it; metadata carries handle + shape + dtype; fallback to shared memory for non-tensor payloads. Mori's intra-node `xgmi` path shows the pattern for AMD; this covers single-GPU NVIDIA co-location (Jetson, single-card servers). +- Acceptance: per-chunk hop latency measured with the WP0 harness drops for the Talker → Code2Wav edge; no change in audio; L1 unit test with a fake two-process exchange. + +### WP8 — Benchmarks and CI lanes (ongoing) + +- `benchmarks/tts/stream_latency_bench.py` and `init_profile.py` results committed per profile; a `cpu` Buildkite lane running `pytest -m "core_model and cpu"` for WP3; a nightly `edge` job for the L3 e2e tests on CUDA; recipes under `recipes/edge/`. + +## 2. Order and effort + +| Order | WP | Effort | Depends on | Needs hardware / permission | +|---|---|---|---|---| +| 1 | WP0 | 2–3 d | — | uv env install (permission); GPU via scheduler | +| 2 | WP1 | 3–5 d | WP0 | GPU | +| 3 | WP2 | 1–2 w | WP0 | GPU | +| 4 | WP3 | 3–5 w | WP0 | none for x86 CPU; aarch64 box later | +| 5 | WP5 | 2–3 w | WP1 | GPU; ExecuTorch install for the `.pte` client (permission) | +| 6 | WP6 | 1–2 w | WP3 | none | +| 7 | WP4 | 2–3 w | WP1, WP2 | Jetson board (blocker today) | +| 8 | WP7 | 2 w | WP1 | GPU | +| 9 | WP8 | ongoing | all | CI access | + +Critical path for a demonstrable result: WP0 → WP1 → WP2 gives a measurably better edge-server tier on the hardware we have; WP3 is the first item that reaches devices without a discrete GPU and is the largest. + +## 3. Risks + +- **vLLM drift**: vLLM-Omni imports 284 `vllm.*` modules and monkeypatches request/output classes; the CPU worker composition in WP3 depends on `CPUWorker`/`CPUModelRunner` internals that changed between the installed 0.28.0 and this checkout's target. Pin the vLLM commit in the branch's `requirements/` and re-validate at each vLLM minor. +- **Upstream acceptance**: each WP is written as an upstream-shaped PR (tests at L1/L3, precheck skill), but WP3 and WP7 are large; expect design review through the repo's `imdesign`/`review-pr` flow before implementation. +- **Model coverage**: the edge profiles are validated per family; families relying on TensorRT (CosyVoice3) or external packages (Fish `fish-speech`) stay CUDA-only until their CPU fallbacks are verified. +- **Hardware**: WP4 and the aarch64 half of WP3 cannot start without boards; WP3-x86 and WP5 can. + +## 4. Definition of done for the branch + +- Edge profiles, CPU platform, and codec-token streaming merged on `edge/omni-edge`, each with L1 tests green locally and L3 e2e green on the CUDA lane. +- Published benchmark JSON for: Qwen3-TTS 0.6B/1.7B on L20X (default vs edge profile), 0.6B on x86 CPU and on one aarch64 box, and (if a board arrives) Jetson. +- Docs: `docs/getting_started/installation/cpu.md`, `recipes/edge/*.md`, and an updated `docs/design/feature/async_chunk.md` section on ramp defaults. + +## 5. Implementation status (branch `edge/omni-edge`, 2026-09-09) + +The branch lives in `/data/zhoutaichang/embedding_infer/vllm-omni` (from `main@7be014bc`, one commit per work package, not pushed). Everything below was measured on this host (one NVIDIA L20X at a time through `gpu run --gpus 1`, or CPU-only on NUMA node 1); the JSON is committed under `benchmarks/tts/edge_results/` and rendered in [experiments/edge_branch/results_summary.md](experiments/edge_branch/results_summary.md). Section numbers refer to [experiments.md §8](experiments.md#8-edge-branch-experiments-2026-09-09). + +| WP | Status | Where | Result / decision | +|---|---|---|---| +| WP0 baseline harness | done | `benchmarks/tts/stream_latency_{bench,metrics}.py`, `tests/benchmarks/` | 1.7B default: TTFA p50 32 ms, playback-start 116 ms, stall 85 ms, RTF 0.10; 0.6B: 28 / 95 / 67 ms. Reproduces the study's ≈80 ms first-chunk gap. | +| WP1 edge profile + `deploy_profile` | done | `vllm_omni/deploy/edge/qwen3_tts.yaml`, `entrypoints/utils.py`, `--deploy-profile` | Ramp `[2,4,8,16,25]` + small batches: stall 0 ms, playback-start = TTFA (44 ms for 1.7B, 41 ms for 0.6B), RTF unchanged. TTFA +12 ms because the first chunk is 2 frames. | +| WP2 init timeline + cold start | done | `engine/init_timeline.py`, `benchmarks/tts/init_profile.py` | Warm 1.7B: default 152 s, eager stage 1 150 s, eager both 111 s (first TTFA 15 s, rejected), `parallel_stage_init` 94.6 s (adopted via the profile's `orchestrator:` block), cold caches 355 s. Dominant phases: engine-core init 51–75 s per stage (model load 9–13 s, talker-MTP graph capture 27 s, predictor compile 12 s, code2wav graphs 4–6 s); the ≤30 s warm target is not met on this stack, the remainder is vLLM's own per-stage init. | +| WP3 CPU platform | done (no vLLM change needed) | `vllm_omni/platforms/cpu/`, `platforms: cpu:` overlay, `docs/getting_started/installation/cpu.md` | 0.6B end to end on the `+cpu` wheel. Thread sweep: 8 threads TTFA 426 ms / RTF 2.8; 16: 292 ms / 1.8; 16 + inductor: 234 ms / 1.4; 32: 252 ms / 1.7. Attribution: the talker step is 136–250 ms per frame (170–310 % of the 80 ms budget); scheduler + hop overhead < 1 %. Not real-time on this CPU class. | +| WP4 hardware adaptation | done (emulated) | `vllm_omni/edge/{hardware_probe,adapt,calibrate,auto_profile,decoder_export}.py`, `deploy/edge/hardware/*.yaml`, `benchmarks/tts/hw_emulation.py` | `deploy_profile="auto"` classifies and sizes every emulated class; Jetson-8 GB emulation (eager, absolute KV budget) TTFA 181 ms / RTF 0.57, Jetson-32 GB 68 ms / 0.09, 8 GB discrete 47 ms / 0.09; CPU classes as in WP3. Decoder export: windowed `torch.export` verified (exact vs eager, 460 MB fp32 per chunk size); ExecuTorch lowering written but not executed (install permission-gated). | +| WP5 codec-token streaming | done (parity partly) | `deploy/qwen3_tts_talker_only.yaml`, `entrypoints/openai/codec_stream.py`, `serving_speech{,_stream}.py`, client + parity scripts, `docs/serving/codec_token_streaming.md` | Streams per step after three runtime fixes found by measurement (FINAL_ONLY coercion, list payloads, delta rows): client TTFA 152–156 ms, stall 0, ramp 2/4/8/16/25 frames, client RTF 0.5 with CPU decode. Client decoder == exact decoder (72–76 dB); end-to-end SNR vs the two-stage server is not meaningful because the two deployments sample different tokens for the same seed (§8.6). | +| WP6 import slimming | done | lazy diffusion imports, `requirements/{core,media,*-edge}.txt`, `VLLM_OMNI_EDGE_BUILD` | AR/TTS path no longer imports diffusers/x_transformers/whisper (guard test); the remaining ≈20 s import overhead is transformers itself. | +| WP7 CUDA-IPC connector | done, not adopted | `connectors/cuda_ipc_connector.py`, `deploy/qwen3_tts_cuda_ipc.yaml` | Talker→code2wav put→get hop: 2.4 ms with shared memory vs 2.9 ms (p95 10.7 ms) with CUDA-IPC (§8.7); below the ≥1 ms gain gate, so the edge profile keeps `SharedMemoryConnector`. | +| WP8 edge scheduling | attribution done; gated items not triggered | `vllm_omni/edge/step_stats.py`, `vllm_omni/edge/scheduling.py`, `--step-stats` | Per-step counters show scheduler + chunk hop + dispatch < 2 % of the frame budget on GPU and CPU; the model step is the cost. Event-driven hand-off, inline AR stage and deadline co-scheduling stay unimplemented by the plan's 5 % gate; the no-preemption admission rule is implemented (exact KV bytes/token) and tested. | + +**Decisions recorded.** + +- *Init knobs*: keep CUDA graphs on both stages; enable `parallel_stage_init` in the edge profile (run 20260909-112200). Eager stage 1 buys nothing; eager on both cuts 41 s of init but costs 15 s of first-request latency. +- *Scheduling*: no scheduler replacement. The dead `connector_get_sleep_s` knob (no consumer in the tree) and the 1 ms receive-loop back-off were confirmed; polling costs ≤1 ms per chunk, so an event-driven hand-off would recover ≤1.3 % of the frame budget and was not built. The admission rule (`vllm_omni/edge/scheduling.py`) is used by the hardware adaptation to size `max_num_seqs` so vLLM never preempts by recompute. +- *Native rewrite*: not warranted by measurement. On the L20X the whole non-model overhead per frame is a few milliseconds; on CPU the talker forward alone is 2–3× the frame budget. The escape hatch remains the C++ S3 orchestrator on llama.cpp/MNN/ExecuTorch for devices where vLLM cannot run at all (phones), which the codec-token stream and the decoder export now serve. +- *CUDA-IPC*: measured no gain on the talker → code2wav edge (the chunk tensor is a small host tensor; the IPC export/ack adds 0.5 ms and a worse tail), so the connector only pays off when a hop carries large device tensors (Qwen3-Omni hidden states); it stays available behind `qwen3_tts_cuda_ipc.yaml`. + +**Not done / deviations.** ExecuTorch was not installed (permission gate; the export path is unit-tested with a fake decoder and the `.pt2` export with the real one). The torch-profiler attribution on CPU was replaced by the step counters (same question, lower overhead). No real edge hardware was used; Jetson/ARM numbers are emulations of the derivation logic, not of the silicon. The customvoice regression e2e passed; the codec-stream e2e passed after the client fix, and three further fixes were needed before frames actually streamed per step (§8.6) — the e2e test checks frames and decoding, not cadence, which is a gap to close. diff --git a/docs/analysis/experiments.md b/docs/analysis/experiments.md new file mode 100644 index 000000000..01ea5a313 --- /dev/null +++ b/docs/analysis/experiments.md @@ -0,0 +1,294 @@ +# Experiments log + +All runs recorded here were executed on 2026-09-08 from this session. Nothing was installed, downloaded, or modified in any checkout. Builds and outputs live in a session scratch directory; logs that matter are copied under [experiments/](experiments/). Numbers below are **measured on this specific host** and are labelled as such; none of them is an edge-device number (the host is an x86 server), so they only calibrate relative costs and validate the execution paths that the design documents rely on. + +## 0. GPU scheduling procedure followed + +Source: `/home/zhoutaichang/feature/GPU调度工具快速上手.pdf` (read in full, 3 pages) and the copy of its rules in `/data/zhoutaichang/CLAUDE.md`. The relevant rules, all honored: + +| Rule (from the guide) | How it was applied | +|---|---| +| Use a personal user, never root | All commands ran as `zhoutaichang` (uid 2005). | +| `gpu status` before starting | Run before submission; all 8 L20X cards were AVAILABLE; a pending booking `84212f67` (another user, GPUs 0-3, 03:00-06:00 daily CI) was noted and avoided by not requesting specific card IDs. | +| Prefer `gpu run --gpus N -- cmd` (auto reserve, sets `CUDA_VISIBLE_DEVICES`, auto release) | Used for all three GPU submissions (runs 1–3). | +| Always `--timeout` on long jobs | `--timeout 30m`. | +| Add `--note` | `--note "tzhouam(claude): qwen3-tts-1.7B streaming TTFA baseline, 12 prompts"`. | +| Release immediately when done | `gpu run` auto-releases; `gpu release` was additionally invoked after the job and `gpu status` re-checked (see §2.4). | +| Never bypass the reservation | No process touched a GPU outside `gpu run`; CPU experiments used no GPU (`GGML_CUDA=OFF`). | + +No dependency installation, model download (`HF_HUB_OFFLINE=1` was exported for the GPU job), or source change was made. Experiments judged expensive (multi-GPU, large models, long sweeps) were **deferred** and are listed in §5 with what permission they need. + +## 1. Hardware and software environment + +| Item | Value (verified with `lscpu`, `free`, `nvidia-smi`, `python -c`) | +|---|---| +| Host | `dedicated-developjob-8gpu2-a029z-…`, Ubuntu 22.04.5, kernel 5.10.134 | +| CPU | Intel Xeon Platinum 8480C, 224 hardware threads, x86_64 (AVX-512 class; `-march=native` build) | +| RAM | 2014 GB total | +| GPU | 8 × NVIDIA L20X, 143,771 MiB each, driver 570.133.20, CUDA 13.0 (`/usr/local/cuda/bin/nvcc`, not on `PATH`) | +| Toolchain | gcc 11.4.0, cmake 4.4.0, ninja 1.13 | +| Python | 3.12.13; `torch 2.13.0+cu129`, `transformers 5.14.1`, `vllm 0.28.0` (wheel), `vllm-omni 0.28.0rc2.dev22+gbe335a86f` (editable, from `/home/zhoutaichang/feature/vllm-omni-main`), `onnxruntime 1.29.0`, `torchao 0.9.0`, `gguf` importable | +| Analyzed commits | see [repository_inventory.md](repository_inventory.md) | + +**Important mismatch.** The GPU experiment runs the *installed* vLLM-Omni (`be335a86f`) against the example script from the *analyzed* checkout (`7be014bc`). The two copies of `examples/offline_inference/text_to_speech/qwen3_tts/end2end.py` were confirmed byte-identical (`diff -q`), and the installed checkout's working tree is clean, but the library code differs by an unknown number of commits. Treat the TTS numbers as "vLLM-Omni around early Sept 2026", not as the exact analyzed commit. + +## 2. Experiment A — vLLM-Omni Qwen3-TTS streaming baseline (GPU, scheduled) + +**Purpose.** Establish a server-class reference for first-audio latency (TTFA), inter-chunk cadence, and end-to-end time of the two-stage talker → code2wav pipeline that [tts_edge_design.md](tts_edge_design.md) proposes to re-host on edge devices. Also verifies the `async_chunk` shared-memory streaming path described in [engines/vllm_omni.md](engines/vllm_omni.md). + +**Model.** `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` (already present in the HF cache, 4.3 GB; the example hard-codes the 1.7B variants, see `_build_inputs` in [`end2end.py`](../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/end2end.py)). Deploy config: the model's default [`vllm_omni/deploy/qwen3_tts.yaml`](../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml) (`async_chunk: true`, `SharedMemoryConnector` with `codec_streaming: true`, `initial_codec_chunk_frames: 1`, `codec_chunk_frames: 25`, stage 0 `gpu_memory_utilization: 0.3`, both stages on logical device `"0"`). + +**Command (run 1, exact; runs 2 and 3 differ as listed in the run-history table below, run 3 invoking [tts_stream_bench.py](experiments/vllm_omni_tts/tts_stream_bench.py) with the same flags plus `--init-timeout 1500 --stage-init-timeout 1500`).** +```bash +export HF_HUB_OFFLINE=1 +cd /exp_tts +gpu run --gpus 1 --timeout 30m \ + --note "tzhouam(claude): qwen3-tts-1.7B streaming TTFA baseline, 12 prompts" -- \ + python /data/zhoutaichang/embedding_infer/vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \ + --query-type CustomVoice --streaming \ + --txt-prompts /data/zhoutaichang/embedding_infer/vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt \ + --output-dir /exp_tts/out --log-dir /exp_tts/logs +gpu release +``` +Prompts: the 12 English sentences shipped in [`benchmark_prompts.txt`](../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/benchmark_prompts.txt). Requests are issued sequentially by the example (`main_streaming` loops over prompts one at a time), so this measures single-stream latency, not throughput. + +**Job record.** + +| Field | Value | +|---|---| +| Scheduler | `canhazgpu` via `gpu run`; type RUN | +| Physical GPU assigned | 6 (log line `Stage 0 logical-to-physical device mapping: 0->6`) | +| Main PID | 2197539 | +| Start | see [experiments/vllm_omni_tts/run1_start.txt](experiments/vllm_omni_tts/run1_start.txt) | +| Full stdout/stderr | [experiments/vllm_omni_tts/run1_stdout.log](experiments/vllm_omni_tts/run1_stdout.log) | + +**Run history (three submissions, all through `gpu run`).** + +| Run | Command variant | Outcome | Log | +|---|---|---|---| +| 1 | example `end2end.py --streaming`, default timeouts | **Failed at startup**: `TimeoutError: Orchestrator did not become ready within 300s`. Stage 0 was still in a cold `torch.compile` of the code predictor (talker CUDA graphs done at 00:13:12, predictor warmup at 00:14:22, timeout at 00:14:34). GPU 6 released by `gpu run`; no stray process. | [run1_stdout.log](experiments/vllm_omni_tts/run1_stdout.log) | +| 2 | same + `--init-timeout 1500 --stage-init-timeout 1500` | Completed (exit 0), 12 WAVs written, engine ready after 187.7 s (stage 0 startup 00:15:33→00:18:22, stage 1 00:18:22→00:19:22). But the example's per-chunk `logger.info` lines are not emitted (module logger with no handler), and in streaming mode `_save_wav` receives only the final message's audio, so the saved WAVs (0.08–1.92 s) are the last partial chunk, not the utterance. No latency data usable. | [run2_stdout.log](experiments/vllm_omni_tts/run2_stdout.log) | +| 3 | session driver [tts_stream_bench.py](experiments/vllm_omni_tts/tts_stream_bench.py) (imports the example's `_build_inputs`/`parse_args`, same flags, adds one warm-up request, records every chunk) | Completed (exit 0); engine ready after 156.8 s (compile caches warm). Results below. | [run3_stdout.log](experiments/vllm_omni_tts/run3_stdout.log), [bench_results.json](experiments/vllm_omni_tts/bench_results.json) | + +**Results (run 3; 1 × L20X assigned by the scheduler; both stages co-located on it; installed vLLM-Omni `be335a86f` + vLLM 0.28.0; `HF_HUB_OFFLINE=1`).** The warm-up request (`w1`: TTFA 420.8 ms, first request after init) is excluded from the statistics. +| Metric (12 sequential requests, batch size 1) | p50 | p95 | max | +|---|---|---|---| +| TTFA, ms (request submit → first audio chunk received in the client coroutine) | 27.7 | 30.7 | 32.8 | +| Inter-chunk gap for 25-frame chunks, ms | 159.9 | 165.4 | 221.7 | +| RTF using streamed chunk audio only | 0.098 | 0.113 | 0.122 | +| RTF using streamed chunks + final-message tail | 0.083 | 0.085 | 0.086 | + +Per-request detail (chunk arrival times in ms after submit; samples at 24 kHz mono): + +| Req | Text (truncated) | TTFA (ms) | Chunk arrivals (ms) / samples | Streamed audio (s) | Final-message tail (s) | Wall (s) | RTF (streamed) | RTF (streamed+tail) | +|---|---|---|---|---|---|---|---|---| +| 0 | Hello, welcome to the voice synthesis … | 29.7 | 30/1920 190/48000 348/48000 | 4.08 | 0.40 | 0.380 | 0.093 | 0.085 | +| 1 | She said she would be here by noon, bu… | 30.6 | 31/1920 195/48000 353/48000 | 4.08 | 0.32 | 0.376 | 0.092 | 0.085 | +| 2 | The quick brown fox jumps over the laz… | 32.8 | 33/1920 190/48000 351/48000 | 4.08 | 0.08 | 0.360 | 0.088 | 0.086 | +| 3 | I can't believe how beautiful the suns… | 27.8 | 28/1920 187/48000 350/48000 510/48000 | 6.08 | 0.16 | 0.518 | 0.085 | 0.083 | +| 4 | Please remember to bring your identifi… | 26.5 | 27/1920 190/48000 351/48000 507/48000 | 6.08 | 0.40 | 0.536 | 0.088 | 0.083 | +| 5 | Have you ever wondered what it would b… | 30.7 | 31/1920 193/48000 353/48000 | 4.08 | 1.92 | 0.497 | 0.122 | 0.083 | +| 6 | The restaurant on the corner serves th… | 25.5 | 26/1920 191/48000 347/48000 | 4.08 | 1.20 | 0.436 | 0.107 | 0.083 | +| 7 | After the meeting, we should discuss t… | 27.5 | 28/1920 187/48000 346/48000 | 4.08 | 1.44 | 0.459 | 0.113 | 0.083 | +| 8 | Learning a new language takes patience… | 27.4 | 27/1920 188/48000 345/48000 503/48000 | 6.08 | 1.60 | 0.623 | 0.103 | 0.081 | +| 9 | The train leaves at half past seven, s… | 26.4 | 26/1920 189/48000 350/48000 503/48000 | 6.08 | 0.80 | 0.565 | 0.093 | 0.082 | +| 10 | Could you please turn down the music a… | 28.5 | 28/1920 250/48000 355/48000 | 4.08 | 1.28 | 0.450 | 0.110 | 0.084 | +| 11 | It was a dark and stormy night when th… | 27.6 | 28/1920 190/48000 348/48000 | 4.08 | 1.20 | 0.437 | 0.107 | 0.083 | + +How to read these numbers: + +- **Chunk structure matches the deploy config.** Every request delivers a first chunk of 1,920 samples (= 1 codec frame at 12.5 Hz, 80 ms of audio; `initial_codec_chunk_frames: 1`), then 48,000-sample chunks (25 frames = 2.0 s; `codec_chunk_frames: 25`). The streamed chunk count is therefore quantized to 1 + 25k frames, and the remainder arrives in the final message (0.08–1.92 s). The "streamed + tail" column is the best estimate of the true utterance length [Inferred: the tail was not verified to be disjoint from the last streamed chunk, but its RMS envelope is speech-like and its length varies per sentence]. +- **TTFA of about 28 ms** covers talker prefill of the CustomVoice prompt, one talker decode step with the 15-codebook code predictor, code2wav of one frame, shared-memory transfer, and the async return path. This is a single-request, warm-graph number on a data-center GPU and is the floor the edge design has to be compared against, not a target for edge. +- **Steady-state cadence is about 160 ms per 25 frames**, i.e. roughly 6.4 ms per codec frame for talker decode + code predictor + code2wav together, well under the 80 ms real-time budget per frame (RTF ≈ 0.08–0.10). +- **Startup dominates**: 157–188 s of engine init (two vLLM engine processes, CUDA-graph capture, `torch.compile`), versus 0.4–0.6 s per utterance. This is the single most important qualitative fact for edge: the vLLM-Omni process model is built for long-running servers, not for cold-start on a device. +- Memory: stage 0 requested 30 % of the card (41.94 GiB budget by config, not need); actual model footprint was not isolated in this run [Unknown]. + +Artifacts kept: the JSON above and the logs. The 13 WAV files and the run-2 WAVs stay in the session scratch directory (not copied, ~2 MB). + + +## 3. Experiment B — llama.cpp CPU-only edge-native baseline (no GPU) + +**Purpose.** Validate the "edge-native baseline" path that [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) compares against: HF checkpoint → GGUF → quantize → CPU inference, for a small VLM whose shape (SigLIP vision encoder + ~400M LLM) is representative of a VLA perception/language backbone. Also measure how much the vision encoder costs relative to token decode, which drives the VLA stage-placement argument in [vla_edge_design.md](vla_edge_design.md). + +**Model.** `HuggingFaceTB/SmolVLM2-500M-Instruct` (cached, 1.9 GB safetensors). Chosen because it was the only cached checkpoint with weights that llama.cpp's converter supports for both text and vision (`conversion/smolvlm.py`, `tools/mtmd/mtmd-image.cpp::mtmd_image_preprocessor_idefics3`). + +**Build (out-of-tree, checkout untouched).** +```bash +cmake -S llama.cpp -B /llama_build -DCMAKE_BUILD_TYPE=Release \ + -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DGGML_NATIVE=ON +cmake --build /llama_build -j 32 \ + --target llama-cli llama-bench llama-simple llama-mtmd-cli llama-tts llama-quantize llama-server +``` +Configure log: [experiments/llama_cpp_cpu/llama_cmake_configure.log](experiments/llama_cpp_cpu/llama_cmake_configure.log) (`GGML_SYSTEM_ARCH: x86`, CPU backend variant `-march=native`). Build finished with exit 0 ([tail](experiments/llama_cpp_cpu/llama_cmake_build.tail.log)). Resulting shared libraries: `libggml-base.so` 0.92 MB, `libggml-cpu.so` 1.53 MB, `libggml.so` 0.06 MB; `llama-cli` executable 1.3 MB (dynamically linked, x86 with AVX-512 code paths, so an ARM/Android build will differ). + +**Conversion and quantization (CPU).** +```bash +PYTHONPATH=llama.cpp/gguf-py python3 llama.cpp/convert_hf_to_gguf.py --outfile smolvlm2-500m-f16.gguf --outtype f16 +PYTHONPATH=llama.cpp/gguf-py python3 llama.cpp/convert_hf_to_gguf.py --outfile smolvlm2-500m-mmproj-f16.gguf --outtype f16 --mmproj +llama-quantize smolvlm2-500m-f16.gguf smolvlm2-500m-q8_0.gguf Q8_0 +llama-quantize smolvlm2-500m-f16.gguf smolvlm2-500m-q4_0.gguf Q4_0 +``` +Logs: [convert_text.log](experiments/llama_cpp_cpu/convert_text.log), [convert_mmproj.log](experiments/llama_cpp_cpu/convert_mmproj.log), [quantize_q8_0.log](experiments/llama_cpp_cpu/quantize_q8_0.log), [quantize_q4_0.log](experiments/llama_cpp_cpu/quantize_q4_0.log). All exit 0. + +| Artifact | Size (bytes) | Note | +|---|---|---| +| `smolvlm2-500m-f16.gguf` | 820,421,728 | text decoder, 409.25 M params as reported by `llama-bench` (the vision tower is excluded) | +| `smolvlm2-500m-q8_0.gguf` | 436,805,728 | | +| `smolvlm2-500m-q4_0.gguf` | 255,864,928 | | +| `smolvlm2-500m-mmproj-f16.gguf` | 199,467,680 | SigLIP vision encoder + projector, always f16 here | + +**Text decode throughput (`llama-bench`, `-p 128 -n 64 -r 3`, CPU backend only, build `e71b80510`).** Raw output: [bench_results.md](experiments/llama_cpp_cpu/bench_results.md). Thread counts 4 and 8 were chosen to mimic a mobile big-core cluster; absolute numbers on a Xeon 8480C are **not** transferable to ARM, only the ratios are indicative. + +| Quant | Threads | Prompt eval pp128 (tok/s) | Generation tg64 (tok/s) | +|---|---|---|---| +| F16 | 4 | 379.6 ± 0.5 | 48.2 ± 4.5 | +| Q8_0 | 4 | 997.3 ± 6.9 | 74.0 ± 13.3 | +| Q4_0 | 4 | 935.4 ± 7.7 | 135.6 ± 1.3 | +| F16 | 8 | 592.7 ± 1.2 | 68.8 ± 7.1 | +| Q8_0 | 8 | 1684.6 ± 73.1 | 150.1 ± 21.0 | +| Q4_0 | 8 | 1522.2 ± 165.3 | 220.1 ± 28.3 | + +Observations (host-specific): decode is memory-bound, so Q4_0 gives ~2.8-3.2× the F16 decode rate; prompt processing is compute-bound and saturates at Q8_0. This is a backbone-only microbenchmark: a Qwen3-TTS frame additionally runs 15 code-predictor sub-steps (`qwen3_code_predictor.py`) and the speech decoder, neither of which is measured here, so it bounds only the talker backbone's share of an 80 ms frame budget and supports no real-time claim for any device. + +**Vision-language path (`llama-mtmd-cli`, Q8_0 text + F16 mmproj, 8 threads, greedy, 48 new tokens).** +```bash +llama-mtmd-cli -m smolvlm2-500m-q8_0.gguf --mmproj smolvlm2-500m-mmproj-f16.gguf \ + --image MNN/resource/images/cat.jpg -p "Describe this image in one sentence." -t 8 -n 48 --temp 0 +``` +Output (verbatim from [mtmd_run.log](experiments/llama_cpp_cpu/mtmd_run.log)): "A small orange kitten is sitting on a rock outdoors, looking directly at the camera." Exit 0. The image is `MNN/resource/images/cat.jpg` (a repo asset, used only as input). + +From the timestamps in [mtmd_run.err](experiments/llama_cpp_cpu/mtmd_run.err): the Idefics3 preprocessor split the image into 13 image chunks (27 prompt chunks in total, image and text interleaved); each image chunk was encoded in 596-1184 ms at 8 threads, i.e. roughly **9-11 s of vision encoding per image** with default tiling. With `--image-max-tokens 64` ([mtmd_run_maxtok64.err](experiments/llama_cpp_cpu/mtmd_run_maxtok64.err)) the preprocessor produced 2 image chunks of ~640 ms each (~1.3 s). Implication for VLA (see [vla_edge_design.md](vla_edge_design.md)): the tiling policy of the image preprocessor, not the LLM, dominates per-frame cost on CPU; an edge VLA must pin a single low-resolution tile per camera, and on CPU-only targets even one SigLIP-B/16-class tile at 512 px costs ~0.6 s at 8 x86 threads in llama.cpp's f16 path, which alone exceeds a 10 Hz control budget. This is a measured fact about this host and this engine, not a projection for ARM or GPU. + +## 4. Experiments not run and why + +| Experiment | Status | Reason / prerequisite | +|---|---|---| +| vLLM-Omni VLA (DreamZero / GR00T / InternVLA-A1) control-loop latency on GPU | **Deferred** | Weights are not in the cache (`GEAR-Dreams/DreamZero-DROID` has metadata only); a download of tens of GB requires permission. Also the DreamZero example is a client that expects a running server. | +| vLLM-Omni Qwen3-TTS 0.6B vs 1.7B TTFA comparison | Deferred | 0.6B weights are cached, but the example hard-codes 1.7B; running 0.6B needs a small script change (source change in an example → permission) or a custom driver. Cheap once approved. | +| vLLM CPU backend (`VLLM_TARGET_DEVICE=cpu`) on x86 | Deferred | The installed vLLM is the CUDA wheel; a CPU build requires a new environment (dependency installation). | +| ExecuTorch export of a TTS/VLA sub-module to `.pte` + XNNPACK run | Deferred | ExecuTorch is not installed and its submodules are uninitialized; `./install_executorch.sh` is a dependency installation. | +| MNN `llmexport.py` + `llm_demo` CPU run | Deferred | Needs an MNN build (cheap) plus its Python export deps; kept for the roadmap's phase 0 gate. | +| ncnn Vulkan build | Deferred | `glslang` submodule uninitialized; CPU-only build possible but there is no cached model in ncnn format. | +| Any ARM / Apple Silicon / Android / Jetson measurement | **Blocked** | No such hardware is reachable from this host. | + +## 5. Reproducible benchmark protocols proposed for the edge work + +These are proposals (not executed) that the roadmap's acceptance criteria refer to; they reuse only artifacts shown to exist above. + +**P-TTS-1 (server reference).** Experiment A as-is, plus `--batch-size 1..8` with `--use-batch-sample`, reporting TTFA p50/p95, inter-chunk p95, RTF = generation wall time / audio seconds, and peak `nvidia-smi` memory. Acceptance for any edge port: same text set, same speaker, TTFA and RTF reported with the identical definitions, audio compared with a speech-recognition WER check on the 12 sentences. + +**P-TTS-2 (edge talker).** Talker exported alone (see [tts_edge_design.md](tts_edge_design.md) for the split); decode tokens/s measured by the target engine's bench tool (`llama-bench`, MNN `llm_bench`, ExecuTorch `llama_main` timing) at 4 and 8 threads; must exceed 12.5 codec frames/s × (1 + margin) for the "real-time" gate. + +**P-VLA-1 (perception cost).** Experiment B's `llama-mtmd-cli` run with `--image-max-tokens` swept over {64, 256, default}, plus the same encoder exported to ExecuTorch/MNN, reporting ms per frame at fixed resolution; gate: encoder + fusion + action head per control step below the chosen control period with p95 jitter below 20 % of it, measured in simulation first. + +## 6. Cleanup + +- GPU: `gpu run` released the reservation after each of the three runs; `gpu release` was run again after runs 2 and 3 and `gpu status --no-schedule` showed no reservation held by `zhoutaichang` ([cleanup_status.txt](experiments/vllm_omni_tts/cleanup_status.txt), [cleanup_status3.txt](experiments/vllm_omni_tts/cleanup_status3.txt)); `nvidia-smi` showed 0 MiB used on all eight cards afterwards. +- Build trees, GGUF files (about 1.7 GB), and generated WAV files live only in the session scratch directory and are not part of the repository; the logs copied under [experiments/](experiments/) total a few hundred KB. +- No checkout was modified (`git status --porcelain` empty for all six after the runs; see [repository_inventory.md](repository_inventory.md)). + +## 7. Reference verification + +All Markdown files under `analysis/` were scanned with [experiments/linkcheck.py](experiments/linkcheck.py): every relative link target must exist, and every `#Lnnn` anchor must be within the target file's length. Final pass: 2,677 links checked, 0 broken (three stale line anchors in the model-level report were corrected during the pass). + +## 8. Edge branch experiments (2026-09-09) + +All runs of this section were executed from the `edge/omni-edge` branch of `/data/zhoutaichang/embedding_infer/vllm-omni` with the two approved environments (`../.venvs/omni-cuda`: vllm 0.28.0 CUDA + editable vllm-omni; `../.venvs/omni-cpu`: vllm 0.28.0+cpu wheel). Results are committed JSON under `vllm-omni/benchmarks/tts/edge_results/` and are rendered as tables in [experiments/edge_branch/results_summary.md](experiments/edge_branch/results_summary.md) (regenerate with `python benchmarks/tts/edge_results/summarize.py`). The status per work package is in [edge_branch_plan.md §5](edge_branch_plan.md#5-implementation-status-branch-edgeomni-edge-2026-09-09). + +### 8.1 GPU scheduling procedure + +The rules of §0 were applied unchanged, with the user's constraint of **at most one GPU at a time**: every GPU job was one `gpu run --gpus 1 --timeout 40m --wait 30m --note "tzhouam(claude): ..." -- env HF_HUB_OFFLINE=1 VLLM_WORKER_MULTIPROC_METHOD=spawn ...` invocation, executed **sequentially** by three shell chains ([chain1.sh](experiments/edge_branch/chain1.sh), [chain2.sh](experiments/edge_branch/chain2.sh), [chain3.sh](experiments/edge_branch/chain3.sh); each chain waits for the previous one to exit before it starts). Each chain ran from a detached git worktree snapshot of the branch so that code edits during a run could not change a running job. `gpu release` and `gpu status --no-schedule` followed every job; the job list from `gpu report --days 7` is in §8.9. CPU-only runs (CPU platform sweep, x86/ARM emulation, decoder export) used no GPU (`VLLM_TARGET_DEVICE=cpu`, `CUDA_VISIBLE_DEVICES=` empty, pinned to NUMA node 1 with `taskset`). + +### 8.2 Streaming latency: default vs edge profile (WP0/WP1) + +`benchmarks/tts/stream_latency_bench.py`, 12 prompts × 3 repeats after 1 warm-up, sequential single-stream requests, metrics from `stream_latency_metrics.py` (TTFA; playback-start latency = max over chunks of arrival time minus audio already buffered; stall at TTFA = playback-start − TTFA; RTF = generation wall time / audio seconds). + +| model | deploy | run | init s | TTFA p50 / p90 ms | playback-start p50 ms | stall p50 ms | RTF | peak GPU MiB | +|---|---|---|---|---|---|---|---|---| +| 1.7B | default | 20260909-103504 | 271 | 32.0 / 35.9 | 116.2 | 84.7 | 0.10 | 46272 | +| 1.7B | edge | 20260909-145133 | 149 | 43.7 / 48.2 | 43.7 | 0.0 | 0.10 | 40876 | +| 0.6B | default | 20260909-104204 | 152 | 27.8 / 32.0 | 95.4 | 67.3 | 0.10 | 46846 | +| 0.6B | edge | 20260909-145500 | 144 | 40.7 / 45.2 | 40.7 | 0.0 | 0.10 | 41248 | + +The edge profile removes the first-chunk stall entirely (the ramp's 2-frame first chunk lands before the 4-frame second chunk is needed) for ≈12 ms of TTFA; RTF and peak memory (per-process, both stages on one card) are unchanged within noise. With the profile's `parallel_stage_init` active (chain 2, runs 20260909-152439/152742, `edge_final2`): 1.7B init 87.7 s, TTFA p50 46.4 ms, stall 0; 0.6B init 86.7 s, TTFA p50 54.0 ms (p90 97 ms, one noisy repeat), stall 0 — the init gain (149 → 88 s) with the same steady-state latency. + +### 8.3 Init timeline and cold start (WP2) + +`benchmarks/tts/init_profile.py` (1.7B, one config per subprocess, `VLLM_OMNI_INIT_TIMELINE` phases from orchestrator and workers; run 20260909-112200 warm caches, 20260909-144054 cold caches): + +| config | init s | first-request TTFA ms | engine_core_init s | load_weights s | talker MTP capture s | predictor compile s | code2wav graphs s | +|---|---|---|---|---|---|---|---| +| default | 152.0 | 646 | 66.2 | 8.9 | 27.4 | 11.6 | 3.6 | +| eager_stage1 | 149.9 | 781 | 67.1 | 6.4 | 29.1 | 11.4 | 0 | +| eager_both | 110.9 | 14956 | 27.8 | 6.2 | – | – | 0 | +| parallel_stage_init | 94.6 | 520 | 75.4 | 10.5 | 26.9 | 12.3 | 3.7 | +| edge (profile, serial init) | 145.7 | 512 | 51.4 | 12.7 | 7.2 | 5.1 | 5.8 | +| cold_cache (fresh VLLM/inductor/triton caches) | 355.5 | 465 | – | – | – | – | – | + +Reading: the two stages initialise serially by default and each spends 50–75 s inside vLLM's `EngineCore.__init__` (device init, weight load, KV profiling, graph capture); running them concurrently on the shared card (`parallel_stage_init`) saves 57 s. Disabling graphs on both stages saves 41 s but the first request then pays the talker-MTP and predictor warm-up (15 s), so the edge profile keeps graphs. A cold inductor/Triton cache adds 200 s; the caches must ship with an edge image. + +### 8.4 CPU platform (WP3) and CPU thread sweep + +`benchmarks/tts/cpu_thread_sweep.py` (0.6B CustomVoice, `+cpu` wheel, bf16 with AMX on the 8480C, stage 0 gets 3/4 of the threads, `--step-stats`), run 20260909-145257 on NUMA node 1 while the GPU chains ran on other cores: + +| threads (stage0+stage1) | init s | TTFA p50 ms | playback-start p50 ms | RTF | peak RSS GiB | talker step ms (mean / p95) | scheduler ms per step | chunk hop enqueue→put / get ms | +|---|---|---|---|---|---|---|---|---| +| 8 (6+2) | 112.7 | 426 | 9438 | 2.78 | 13.3 | 250.5 / 499.7 | 0.13 | 1.37 / 0.94 | +| 16 (12+4) | 109.5 | 292 | 5346 | 1.84 | 13.3 | 147.4 / 219.6 | 0.12 | 0.82 / 0.79 | +| 16 + inductor (`VLLM_OMNI_CPU_INDUCTOR=1`) | 103.4 | 234 | 3642 | 1.43 | 13.3 | 138.9 / 180.0 | 0.12 | 0.83 / 0.91 | +| 32 (24+8) | 155.7 | 252 | 4995 | 1.73 | 13.5 | 136.2 / 168.9 | 0.11 | 0.86 / 0.55 | + +The talker step (one 80 ms codec frame per step, including the 15 code-predictor sub-steps) costs 136–250 ms, i.e. 1.7–3.1× the frame budget, and stops scaling beyond 16 threads; inductor gives 6–22 %. Scheduler work is 0.1 ms per step and the shared-memory chunk hop about 1 ms per chunk, together under 2 % of the budget. The earlier `cpu_smoke` run (12+4 threads from the static overlay) is in the summary file (TTFA 292 ms, RTF 1.7). Conclusion unchanged from the study: this CPU class serves offline synthesis, not real-time streaming; a 4-core/AVX2 box (§8.5) is 12× slower than real time. + +### 8.5 Hardware emulation (WP4) + +`benchmarks/tts/hw_emulation.py` with `deploy_profile="auto"` and `VLLM_OMNI_HW_PROFILE` overrides (0.6B, 12 prompts × 1). CPU cases ran in the CPU venv pinned to N cores with `ATEN_CPU_CAPABILITY` downgrades; GPU cases ran on the L20X with the derived absolute KV budgets and eager rules of the emulated class (the card's memory is not capped, so peak GPU MiB shows what the budget admits, not a hard limit). + +| case | class chosen | dtype / threads or budget | init s | TTFA p50 ms | playback-start p50 ms | RTF | peak RSS GiB / GPU MiB | +|---|---|---|---|---|---|---|---| +| x86_cpu_avx2 (4 cores, 8 GiB) | x86_cpu | fp32, 3+1 threads, KV 1.2 GiB | 120 | 2651 | 54419 | 11.9 | 12.8 / – | +| x86_cpu_avx512 (8 cores, 16 GiB) | x86_cpu | 6+2 threads | 109 | 679 | 7874 | 2.45 | 14.4 / – | +| x86_cpu_amx (16 cores, 32 GiB) | x86_cpu | bf16, 12+4 threads | 116 | 513 | 5692 | 1.72 | 23.0 / – | +| arm64_cpu_logic (RK3588-like, logic only) | arm64_cpu | fp32, 3+1 threads, KV 2.2 GiB | 108 | 2183 | 50262 | 11.9 | 13.0 / – | +| jetson_orin_8g (unified 8 GiB) | jetson | eager both, absolute KV budget | 129 | 181 | 196 | 0.57 | – / 6236 | +| jetson_orin_32g (unified 32 GiB) | jetson | graphs on | 183 | 68 | 68 | 0.09 | – / 23376 | +| cuda_discrete_8g (8 GiB VRAM) | cuda_discrete | eager + small KV | 189 | 47 | 47 | 0.09 | – / 9680 | + +The derivation never allowed preemption (checked by `tests/core/test_edge_scheduling.py` against the exact talker KV geometry, 112 KiB per token). The Jetson-8 GB emulation shows the cost of running eager on a small budget (RTF 0.57, TTFA 181 ms) and is the case where the S3 native path should be compared on real hardware. + +### 8.6 Codec-token streaming: cadence and parity (WP5) + +`benchmarks/tts/codec_stream_e2e.py` launches the talker-only server (eager stage 0), streams codec frames to the client decoder (CPU), then launches the full two-stage server (eager stage 0, same seed) for the reference PCM, and reports client TTFA and the length-trimmed SNR. Four runs were needed: + +| run | outcome | root cause / fix | +|---|---|---| +| chain 1 (20260909-151221) | client crashed in `batched_chunked_decode` | the decoder takes a `[B, Q, F]` tensor, not a list; fixed in `536866b9` and verified against the real 0.6B decoder on CPU | +| chain 2 (20260909-154659) | frames decoded, but all codec frames arrived in one message ≈2.1 s after the request (client TTFA 2.2–2.7 s = whole generation), SNR −3 dB | see below | +| chain 4 (20260909-160935) | same, with counters: **1 engine output per request, first codes at output #1** | the speech path forced `FINAL_ONLY` outputs for Qwen3-TTS deployments without async chunks (`qwen3_full_payload`), which a talker-only deployment also is; fixed in `b22dabe2` (talker-only = final stage with latent output keeps DELTA) | +| chain 5 (20260909-162326) | server now emits per step (2 outputs by step 2) but failed: "inhomogeneous shape" in row alignment | in DELTA mode the payload is a list of per-step tensors; normalized in `380e8022` | +| chain 6 (20260909-163326) | 53 engine outputs for 52 rows, yet the client still got one message | each per-step payload is the *newest row only*; the alignment treated it as a cumulative snapshot (one yield at row 1, one at the end); fixed in `c893b638` (`accumulate_codec_rows`, unit-tested with per-step deltas) | +| chain 7 (20260909-164759) | **streams per step**: client TTFA 152–156 ms, playback-start = TTFA, stall 0, chunks 2/4/8/16/25 frames arriving 145 / 285 / 570 / 910 ms apart (≈36 ms per frame from the eager talker-only server + CPU decode), client RTF 0.5–0.6, first utterance 15 s (server warm-up) | – | + +Parity was separated into two questions with `benchmarks/tts/codec_parity_diag.py` on the chain-4 codes (client now saves the received rows): (1) client chunked decode vs the exact non-chunked decode of the same codes: **72.7–75.9 dB SNR, correlation 0.99999999** for all four utterances, so the client-side decoder (ramp schedule, 72-frame context, stateful caches) reproduces the server's `Code2Wav` math; (2) the full server's PCM vs the exact decode of the client's codes: −3 dB, correlation ≈ 0 with identical frame counts, i.e. the two server processes sampled different code sequences for the same text and seed (the `seed` request field reaches both paths, so the divergence is numeric between the talker-only and the two-stage deployments; the speech API exposes no temperature, so a greedy comparison was not possible). The ≥40 dB end-to-end acceptance is therefore **not demonstrated**; decoder parity is, and a token-level tap on the two-stage server (or a greedy sampling switch) is the follow-up. The wire cost is unchanged from the design (16 int16 codebooks + 7-byte header per 80 ms frame ≈ 490 B/s). `tests/e2e/online_serving/test_qwen3_tts_codec_stream.py` passed in chain 2 (frames + decode), before the cadence fixes; it does not assert cadence. + +### 8.7 Per-step attribution on GPU (WP8) and the CUDA-IPC connector hop (WP7) + +`stream_latency_bench.py --step-stats` (0.6B, 12 prompts × 3, chains 2 and 3). Counters per process role; share = mean / 80 ms frame budget (one talker step = one frame). + +| counter | default deploy (shm) | edge profile | cuda_ipc deploy | +|---|---|---|---| +| stage 0 engine-core step (talker + code predictor) | 6.43 ms (p95 10.8) = 8.0 % | 5.96 ms = 7.4 % | 5.79 ms = 7.2 % | +| stage 0 `schedule()` + `update_from_output()` | 0.12 + 0.12 ms = 0.3 % | 0.09 + 0.09 ms = 0.2 % | 0.09 + 0.08 ms = 0.2 % | +| stage 1 engine-core step (code2wav, incl. polling steps) | 0.54 ms | 0.38 ms | 0.35 ms | +| chunk hop: enqueue→put / put / get | 0.62 / 0.48 / 0.68 ms | 0.33 / 0.23 / 0.34 ms | 0.34 / 0.29 / 0.36 ms | +| chunk hop put→get (lock-file mtime → get, chain 3) | 2.42 ms (p95 2.58) = 3.0 % | – | 2.90 ms (p95 10.7) = 3.6 % | + +Everything outside the model step (scheduler, transport, polling) is below 2 % of the frame budget per stage, and the whole hop from talker put to code2wav get is 2.4 ms with shared memory. CUDA-IPC does not improve it (2.9 ms mean, worse tail): on this edge the chunk tensor is already a small host tensor (codes), so the IPC export/ack adds work without removing a copy. **Decision: `qwen3_tts_cuda_ipc.yaml` stays optional; the edge profile keeps `SharedMemoryConnector`.** The orchestrator dispatch counter only fires under the event-driven orchestrator loop (`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`); the default poll loop was used here, so it is absent from the tables. + +### 8.8 Decoder export (WP4, phone/NPU class) + +`python -m vllm_omni.edge.decoder_export --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --ramp 2 25` (CPU, no GPU): two `torch.export` programs (window 74 and 97 frames), exact match against the eager decoder, 460 MB fp32 each (manifest committed as `benchmarks/tts/edge_results/decoder_export_manifest_0.6B_torch_export.json`). ExecuTorch lowering not executed (package not installed; install is permission-gated). + +### 8.9 Job records and cleanup + +See the `gpu report --days 7` extract in [experiments/edge_branch/gpu_report.txt](experiments/edge_branch/gpu_report.txt) (all jobs of this section are the `tzhouam(claude): WP…` notes, type RUN, one card each, none overlapping). `gpu status --no-schedule` showed no reservation by `zhoutaichang` after the last chain; the worktrees `vllm-omni-run*` were removed; venvs stay outside the tree. diff --git a/docs/analysis/implementation_roadmap.md b/docs/analysis/implementation_roadmap.md new file mode 100644 index 000000000..165a27991 --- /dev/null +++ b/docs/analysis/implementation_roadmap.md @@ -0,0 +1,90 @@ +# Implementation roadmap + +Phased plan for the designs in [tts_edge_design.md](tts_edge_design.md) and [vla_edge_design.md](vla_edge_design.md), following the strategy decision in [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) (S4 hybrid first for TTS, S3 lightweight orchestration as the main path, S5 edge-native numbers at every phase, S1 on Jetson as the reference tier). Each phase has entry conditions, deliverables, acceptance criteria, and the reproducible benchmark it must publish. Nothing below has been executed except the Phase 0 items marked "done" in [experiments.md](experiments.md). + +## Phase 0 — Reference numbers and toolchain gates (1–2 weeks) + +Entry: this analysis. No device hardware required for the first half. + +| Item | Status | Acceptance | +|---|---|---| +| Server reference for Qwen3-TTS 1.7B streaming (TTFA, cadence, RTF) | **done** on L20X (p50 TTFA 27.7 ms, RTF ≈ 0.08–0.10) | Numbers and JSON archived; re-runnable with the recorded command | +| CPU edge-native baseline with llama.cpp (build, convert, quantize, bench, VLM run) | **done** on x86 (Q4_0 decode 135.6 tok/s @ 4 threads; SigLIP tile 0.6–1.2 s @ 8 threads) | Same | +| Qwen3-TTS 0.6B server reference | pending (needs a driver that overrides the example's hard-coded 1.7B ids; no source change) | TTFA/RTF for 0.6B on the same prompts | +| llama.cpp `llama-tts` Qwen3-TTS on x86 CPU (needs the GGUF conversion of the cached HF checkpoint; check whether `conversion/` supports it at this commit) | pending | WAV produced; RTF at 4/8 threads; codebook-0 agreement with the server under greedy sampling | +| MNN build with `MNN_BUILD_LLM` + `llmexport.py` of Qwen3-TTS 0.6B + `qwen3_tts_demo` on x86 CPU | pending (requires its Python export dependencies; permission to install) | Same metrics as above | +| ExecuTorch install + export of the Qwen3-TTS decoder (`modeling_qwen3_tts_tokenizer_v2.py`) to `.pte` on XNNPACK | pending (dependency installation; submodules) | Decoder PCM SNR vs torch ≥ 40 dB on recorded codes [Proposal threshold] | +| VLA reference: π0 or GR00T on the L20X through the OpenPI endpoint with recorded observations | pending (weights not cached; download needs permission) | Action chunks for ≥ 100 recorded observations saved as the parity oracle | +| Device procurement / access: one ARM Linux board, one Android phone with Qualcomm NPU, one Apple Silicon device, one Jetson | pending | Devices enumerated with SoC, RAM, OS, and thermal envelope | +| **Checkpoint/task parity gate**: pin one Qwen3-TTS checkpoint and task per engine (server benchmark used CustomVoice 1.7B; llama.cpp README shows 1.7B Base; MNN demo needs reference audio and hard-codes `hiddenSize = 1024`) | pending | A table stating, per engine, which size/task variant is confirmed to run, with the exact converted artifact; comparisons in later phases use only matching rows | +| **Playback-gap measurement**: rerun the server driver with `codec_chunk_ramp` / `codec_chunk_adaptive` enabled and report uninterrupted-playback latency alongside TTFA | pending (cheap, same GPU procedure) | Both metrics in the JSON schema; the default 1→25 schedule's ≈80 ms gap quantified | + +Exit criterion: for each runtime kept in scope, a cached-model artifact and a one-command benchmark script exist and run on x86, so device runs in Phase 1 are only a change of host. + +## Phase 1 — TTS hybrid (S4): server talker, on-device decoder (2–4 weeks) + +Deliverables: + +1. vLLM-Omni side: a codec-token streaming mode. This is a new wire contract, not only an adapter: `serving_speech_stream.py`'s `_generate_and_send` consumes PCM from the full pipeline today, so the deliverable is a talker-only deploy profile plus a stream message type carrying `codes.audio` frames (16 × int16 per frame) with defined frame ordering, EOS, cancel, and decoder-state/revision semantics (integration points in [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) §5). This is the only proposed change inside vLLM-Omni and must go through its review process (`review-pr`/`precheck-pr` conventions in the repo). **Gate**: the hybrid-first ordering holds only if this prototype and the on-device decoder measurement (item 2) both pass; otherwise Phase 2 proceeds first. +2. Device side: `IDecoderAdapter` implementations on llama.cpp (from the mtmd code2wav graph) and ExecuTorch (`.pte` from Phase 0), driven by the same chunk schedule as the server reference (ramp/adaptive from the deploy YAML, 72-frame left context; 1→25 kept as the measured baseline) and `reset()` on abort. +3. A small client that plays audio from the codec stream with a 2 s ring buffer and supports interruption. + +Acceptance: + +- TTFA (client-observed, including network) ≤ server TTFA under the same chunk schedule + RTT + first-chunk decode time; **uninterrupted-playback latency** (earliest start with no later underrun, using the ramped chunk schedule) reported alongside; inter-chunk gap p95 ≤ 100 ms on the device decoder for 25-frame chunks (else reduce steady chunk size). +- Decoder parity: PCM SNR ≥ 40 dB against the server decoder on identical codes; ASR WER on the 12 benchmark prompts within 1 point of server audio. +- Interruption: audio stops within one chunk length (≤ 2 s worst case, target ≤ 200 ms with a 2-frame ring) and the next utterance starts cleanly (no residual decoder state, verified by bit-identical output to a cold decode). + +Benchmark: `P-TTS-1` and `P-TTS-2` from [experiments.md](experiments.md) §5, run on device and on x86, published as JSON in the same schema as `bench_results.json`. + +## Phase 2 — TTS fully on device (S3) on llama.cpp and MNN (4–6 weeks) + +Deliverables: + +1. Edge TTS orchestrator (C++) implementing the loop in [tts_edge_design.md](tts_edge_design.md) §2.2 over `ITalkerAdapter`/`IDecoderAdapter`/`ISpeakerAdapter`. +2. llama.cpp binding: talker + predictor via the existing Qwen3-TTS mtmd graphs, chunked decode, abort check per step; Android JNI and Apple xcframework builds. +3. MNN binding: chunked `speech_decoder` calls with carried state (pattern from the Qwen2.5-Omni streaming path), talker at W4; Android app integration (`MnnLlmChat` has the host UI), QNN delegation of the decoder as an experiment. +4. Parity harness against vLLM-Omni offline outputs (greedy) for talker codes and decoder PCM. + +Acceptance: + +- 0.6B path: RTF ≤ 0.5 on the ARM board at 4 threads and on the phone's big cores; TTFA ≤ 300 ms; resident memory ≤ 2 GB; 10-minute continuous synthesis without thermal throttling below the RTF gate (log SoC temperature). +- 1.7B path: RTF ≤ 1.0 on Apple Silicon Metal and on the phone GPU/NPU; report, do not gate, on CPU. +- Codebook-0 greedy agreement ≥ 95 % on the first 25 frames of each prompt (differences after the first divergence are expected; compare prefix only) [Proposal metric]. +- Interruption and backpressure behave as specified (tests with abort at frame 1, mid-chunk, and after EOS). + +Benchmark: `P-TTS-2` plus the interruption test suite; results per device in a table with the same columns as the server run. + +## Phase 3 — ExecuTorch TTS path for CoreML/QNN and Jetson reference (4 weeks, parallel with Phase 2) + +Deliverables: talker export via `export_llm` (Qwen3 config, 8da4w), predictor and decoder as separate programs, a runner modelled on the Voxtral TTS runner; CoreML and QNN partitioner recipes; S1 reference run of vLLM-Omni on a Jetson (install, cold-start time, TTFA/RTF) for the same prompts. + +Acceptance: same gates as Phase 2 on the ExecuTorch path; the Jetson S1 run documents cold start and memory (expected to fail the cold-start gate; recorded as the reference tier, not as a product path). + +## Phase 4 — VLA parity and simulation (6–8 weeks) + +Deliverables: + +1. Exports of π0 (plain torch, CPU parity test exists in the repo) and GR00T N1.7 (HF wrapper) into ExecuTorch programs: `vision_encoder`, `fusion`, `action_head`; MNN exports of the same via ONNX for Android NPU experiments. +2. `IVlaRunner` implementations and the controller/safety split from [vla_edge_design.md](vla_edge_design.md) §2.1, with the chunk buffer, staleness check, and watchdog. +3. Parity suite: recorded observations → action chunks compared with the OpenPI server (max-abs, per-dimension RMS) at K = 4 (GR00T) and K = 10 (π0). +4. Simulation closed loop (external simulator, LIBERO-style tasks as available for the checkpoints) with the edge runner in the loop. + +Acceptance: + +- Parity: with the flow noise passed explicitly (π0 `sample_actions(noise=...)`) so both sides are deterministic, per-dimension RMS error ≤ 2 % of the action range at fp16 weights, ≤ 5 % at 8da4w [Proposal thresholds, to be revisited after the first measurements]. +- Latency on device: inference wall time p50 ≤ 250 ms and p95 ≤ 400 ms for one 224² camera on the phone NPU or Apple ANE; report the three-camera case; chunk age never exceeds the staleness bound in a 10-minute run [Proposal]. +- Simulation success rate within 5 points of the server policy over ≥ 50 episodes per task. +- Fault injection tests pass (delayed frames, dropped camera, CPU frequency cap) with the safety layer engaging as specified. + +Benchmark: `P-VLA-1` from [experiments.md](experiments.md) §5 for perception cost, plus the parity and simulation suites; publish per-stage ms, jitter, memory, temperature. + +## Phase 5 — Hardware-in-the-loop and hybrid re-planning (open-ended) + +Only after Phase 4 passes: hardware-in-the-loop with an independent safety layer; optional cloud re-planning via vLLM-Omni for task text updates; long-duration soak (thermal, memory growth) on every target. + +## Cross-cutting + +- **Repository conventions**: any change proposed to vLLM-Omni or vLLM follows their `AGENTS.md` (uv environments, human-owned PRs, disclosure of AI assistance); llama.cpp explicitly forbids autonomous agent contributions, so llama.cpp work stays in a private fork or is authored by a human maintainer of this project. +- **Reproducibility**: every benchmark is a script plus a JSON output plus the exact commit of each engine; device runs record SoC, OS, thermal state, and governor settings. +- **Permission gates** (per the task's rules): dependency installs (ExecuTorch, MNN export deps), model downloads (π0/GR00T/DreamZero weights), and any source change in the six checkouts require explicit approval before they are executed. diff --git a/docs/analysis/minicpmo_edge_profile.md b/docs/analysis/minicpmo_edge_profile.md new file mode 100644 index 000000000..88e6118d0 --- /dev/null +++ b/docs/analysis/minicpmo_edge_profile.md @@ -0,0 +1,119 @@ +# MiniCPM-o 4.5 on the edge: measured profile (2026-09-11) + +Companion to [mobile_htp_simulation.md](mobile_htp_simulation.md) (Qwen3-TTS) using the same instruments: vLLM-Omni's +StepStats for per-stage attribution, the Xeon 8480C for CPU, one H200 through `canhazgpu` for GPU, and Qualcomm AI Hub +for real Snapdragon devices. Scripts and raw results: `experiments/edge_engine_compare/minicpm/` and +`experiments/edge_engine_compare/results/minicpmo/`. + +## 1. What this model is, and why the shape matters + +MiniCPM-o 4.5 is a three-stage pipeline in vLLM-Omni (`MINICPMO_4_5_PIPELINE`): an 8 B omni LLM (SigLIP vision tower + +Whisper audio encoder + text backbone) → a TTS head → Code2Wav. Weights are local (19 GB fp16, ≈9.5 B parameters). + +The speech half is much lighter than the thinker, and much lighter than Qwen3-TTS's: + +| component | parameters | rate | +|---|---|---| +| TTS head (Llama-style decoder, 20 layers × 768, 12 heads, no GQA, no q/k-norm) | 194 M | 1 token per 40 ms (25 Hz) | +| Code2Wav flow matching (CFM DiT, Step-Audio2 / CosyVoice2 lineage) | 156 M | 10 denoising steps per chunk | +| Code2Wav HiFT generator | 21 M | 1 chunk = 25 tokens = 1.0 s of audio | +| thinker (vision + audio + text, 8 B) | ≈9.2 B | prompt-rate | + +**`num_vq = 1`**: one codebook, so one decode step per speech token. Qwen3-TTS emits 16 codebooks at 12.5 Hz, which +costs one talker step *plus fifteen* code-predictor sub-steps per 80 ms frame. That single architectural choice is worth +3.5× on a phone (§4). The vocoder is also genuinely streaming already — 3 tokens of lookahead, persistent flow and HiFT +caches — which is the design Qwen3-TTS's 72-frame-context vocoder still lacks. + +## 2. GPU (one H200, 6 concurrent requests, text → speech) + +Deploy `vllm_omni/deploy/minicpmo_4_5.yaml` (all three stages on one card), 6 prompts from `benchmarks/tts/prompts_12.txt`, +`max_tokens` 192 for the thinker and the stage-1 deploy defaults for the TTS head. + +| metric | value | +|---|---| +| engine init (warm caches) | 253 s | +| first text | p50 393 ms, p90 598 ms | +| complete audio per request | p50 5.81 s, p90 7.60 s (offline path yields the whole waveform, so this is completion, not first-chunk latency) | +| batch throughput | 30.2 s of audio in 22.6 s wall → **RTF 0.748** at concurrency 6 | + +Per-stage step attribution (mean over the run): + +| stage | steps | mean step | mean model forward | +|---|---|---|---| +| 0 · thinker | 46 | 9.66 ms | 2.31 ms | +| 1 · TTS head | 274 | 6.53 ms | 1.72 ms | +| 2 · Code2Wav | 469 | 14.88 ms | – | + +The largest single number in the table is not compute: stage 2's `hop.put_to_get` averages 3.1 s, i.e. codec chunks wait +in the shared-memory connector while the vocoder works through its backlog. On one card the vocoder is the queue. + +## 3. CPU (Xeon 8480C, pinned) + +| stage | 8 threads | 16 threads | real-time factor at 16 threads | +|---|---|---|---| +| TTS head, per 40 ms token | 17.1 ms | 11.6 ms | **0.29** | +| Code2Wav, per 1.0 s of audio | 1033 ms | 833 ms | **0.83** | +| one-time speaker setup (x-vector + prompt mel + caches) | 11.7 s | 9.9 s | cacheable per voice | + +Inside the vocoder chunk the split is lopsided, and in the opposite direction from Qwen3-TTS: + +| half | per 1.0 s of audio, 16 threads | share | +|---|---|---| +| flow matching (10-step CFM DiT) | 652 ms | **91 %** | +| HiFT generator | 65 ms | 9 % | + +Qwen3-TTS's vocoder was elementwise-bound (76 % in Snake activations on sample-rate tensors) and therefore wanted a GPU; +MiniCPM-o's is matmul-bound in a DiT and therefore suits an NPU. The two models need opposite placements. + +The step count is a config field (`tts_config.s3_stream_n_timesteps`), and it trades linearly: + +| flow steps | per 1.0 s of audio | mel fidelity vs the 10-step default | +|---|---|---| +| 10 (default) | 542 ms | reference | +| 6 | 341 ms | **30.6 dB** | +| 4 | 236 ms | 16.5 dB | +| 2 | 127 ms | 9.6 dB | + +Six steps is a 1.6× vocoder speed-up at 30.6 dB on the mel spectrogram — worth a listening test, since it costs nothing +but a config change. + +## 4. Real Snapdragon devices (Qualcomm AI Hub) + +The TTS head exported through `vllm_omni/edge/qnn_export.py` (Llama-style path: `qk_norm=False`, weight-normalised code +head), 256-token KV cache, QNN context binary, every operator on the Hexagon NPU. + +| build | Galaxy S25 (8 Elite) | Galaxy S24 (8 Gen 3) | +|---|---|---| +| fp16, per 40 ms token | **7.08 ms** | **8.18 ms** | +| fp16 fidelity vs torch fp32 | 48.9 dB, top-1 token matches | 49.2 dB, top-1 matches | +| int8 weights (w8a16), per token | 4.88 ms | 6.15 ms | +| int8 fidelity | **2.3 dB, top-1 differs** | **0.1 dB, top-1 differs** | + +Per second of speech, the comparison with Qwen3-TTS on the same phone is the headline: + +| model, S25 NPU fp16 | per second of audio | +|---|---| +| **MiniCPM-o 4.5** (1 codebook @ 25 Hz, 194 M) | **177 ms** | +| Qwen3-TTS 0.6B (16 codebooks @ 12.5 Hz: talker 15.8 + predictor 33.9 per 80 ms frame) | 621 ms | + +**Int8 fails here too.** Per-tensor int8 weights destroy MiniCPM-o's code head exactly as they destroy Qwen3-TTS's code +predictor (3.1 dB there, 2.3 dB here) — two unrelated families, same failure, so this is a property of speech-token heads +rather than of one model or one runtime. The faster int8 rows stay timing-only. + +Not yet measured on device: the Code2Wav flow + HiFT (its streaming caches need the same flattening the KV cache needed), +and the 8 B thinker, which at fp16 is 16 GB and does not fit a phone at all — int4 would be mandatory, and §4 is the +reason to expect trouble there. + +## 5. Reading + +- **For a phone, MiniCPM-o's speech stack is the better architecture** and Qwen3-TTS's is the better size. One codebook at + 25 Hz costs 177 ms of NPU time per second of audio against 621 ms; but the thinker in front of it is 8 B, where + Qwen3-TTS's whole pipeline is 0.6 B. +- **Placement is model-specific.** Qwen3-TTS wants transformer-on-NPU + vocoder-on-GPU. MiniCPM-o's vocoder is a DiT, so + the NPU is plausible for both halves; that is the next measurement. +- **The cheapest real win is the flow step count**, not quantization: 10 → 6 steps is 1.6× at 30.6 dB, while every int8 + path measured in this study has been numerically broken. +- Two bugs found while profiling, both in the offline path rather than the model: `examples/offline_inference/minicpmo/end2end.py` + builds two sampling-parameter sets and fails on this three-stage pipeline with `Expected 3 sampling params, got 2`, and + its talker parameters cap the codec LM at `max_tokens=1`, which silently yields zero audio. The driver in + `experiments/edge_engine_compare/minicpm/bench_minicpmo.py` passes three sets and the stage-1 deploy defaults. diff --git a/docs/analysis/optimization_plan.md b/docs/analysis/optimization_plan.md new file mode 100644 index 000000000..09819fe09 --- /dev/null +++ b/docs/analysis/optimization_plan.md @@ -0,0 +1,585 @@ +# Optimization plan: inference and harness + +Written 2026-09-12 against [spark_edge_support.md](spark_edge_support.md), then +revised after an independent review by `gpt-6-astra` through the Codex hook +(read-only against these checkouts; the sandbox could not start because user +namespaces are disabled on this host, so it ran unsandboxed under explicit +read-only instructions and the checkouts were verified clean afterwards — the +same procedure recorded in [README.md](README.md) for the 2026-09-08 study). + +The review rejected the first draft's ordering and several of its expected +gains. Most of that was correct and is applied below; §"Review record" lists +what I accepted, what I corrected with new measurements, and why. + +Every item starts from a measurement in this tree, states the expected gain, +and says how it would be verified — because on this host a claim without a +verification method is how three earlier conclusions turned out to be wrong. +**A fourth is now on the list**: the first draft of this plan mis-described the +740 MB it proposed to remove. + +## Where the time actually goes + +W4A16 g128 decode, 24 threads, per-op profile (reproducible to ~5%, where +throughput swings ~30%). The first draft built this table from two different +builds — 5.91 ms is the **W4A8** GEMM, 6.85 ms the W4A16 one +([support:193](spark_edge_support.md)) — so the step it described (11.8 ms) was +neither configuration. The W4A16 step is **12.51 ms**: + +| | ms | share of 12.51 | +|---|---|---| +| int4 GEMM (tinygemm) | 6.85 | 55% | +| inductor elementwise (norms, rope, GEGLU, residuals) | 2.20 | 18% | +| vLLM attention ops (4 per layer × 28) | 1.31 | 10% | +| custom-op dispatch boundary (W4A16 only) | 0.55 | 4% | +| unattributed remainder (sampling, view/slice churn) | 1.60 | 13% | + +Two corrections to how this table was read. The GEMM runs near this socket's +achievable bandwidth for *its own* shape, which says the GEMM is memory-bound; +it does **not** say the other 45% is framework overhead — attention, softmax, +norms and sampling contain real arithmetic. And the per-step savings of W4A8 +over W4A16 do not add: 0.94 ms (GEMM) + 0.48 ms (dispatch) = 1.42 ms predicted +against 12.51 − 11.55 = **0.96 ms** measured. Something offsets them, and no +item below may assume its saving reaches the step time unchallenged. + +## Where the memory actually goes + +Anonymous RSS by phase ([support:324](spark_edge_support.md)): + +| phase | W4A8 (GPTQ) | W4A16 (tinygemm) | +|---|---|---| +| before import | 12 MB | 12 MB | +| after `import vllm_omni` | 513 MB | 515 MB | +| after engine construction | **7338 MB** | 5023 MB | +| after releasing free arenas | 3510 MB | 3411 MB | + +The first draft treated the post-release figure as the number to reduce. That +is the wrong target for an edge device: **7338 MB is the peak that has to fit**, +and `release_after_load()` runs only after `load_model` and warm-up have both +returned ([patch.py:730](../vllm-omni/vllm_omni/patch.py)). A device that cannot +survive the load never reaches the release. This reordered the plan. + +The draft's weight breakdown ("0.86 GB quantized + 0.54 GB bf16 embedding") is +also wrong for the checkpoint actually measured: `spark2_5-int4-g128/config.json` +sets `quantize_embedding: true` with `tie_word_embeddings: true`, so there is no +bf16 embedding, and the persistent prefill duplicate was left out of the +breakdown entirely. There is therefore no trustworthy ledger to subtract from — +which is why the "unexplained 1.6 GB" item is now a prerequisite, not a +long-shot. + +--- + +## Inference + +### I0. Peak load-time memory — *now first* + +7338 MB peak vs 3510 MB steady is the gap that decides whether a device can run +this at all. The peak comes from holding the checkpoint, the dequantized +intermediate and the repacked layouts alive at once, per layer, with freed +arena pages not yet returned. + +**Do**: (a) release arenas *during* the load, per layer or per N-layer block, +rather than only after `load_model` returns; (b) repack blockwise so no +whole-tensor bf16 intermediate exists; (c) as the durable fix, write the packed +layout to a deployment artifact at build time so the device loads what it runs. + +**Expect**: peak toward the steady 3.4–3.5 GB. Unquantified deliberately — the +mechanism is understood, the size is not. + +**Verify**: sample `RssAnon` on a timer through load (0.2% reproducible), report +the maximum, not the endpoint. The harness measures endpoints today and would +not have caught this. + +### I1. One packed weight layout, not two — *replaces the draft's I2* + +The W4A16 build keeps a second copy costing **740 MB** (3542 → 2802 MB) and +buying **7.3× prefill** (3252 → 448 tok/s). The draft described this as "prefill +unpacks to bf16" and proposed a tiled dequant kernel to avoid the bf16 copy. +That was wrong about the artefact. `_finalize_int4` keeps `weight_halves` — a +**second packed 4-bit copy** — plus cloned `weight_group_scale` and +`weight_group_zero`; the bf16 matrix is already a per-call temporary and +`weight_packed` is deleted +([cpu_int4.py:344,375](../vllm-omni/vllm_omni/model_executor/layers/quantization/cpu_int4.py)). + +So the cheap fix is not a new GEMM. It is to **dequantize directly from the +tinygemm layout** and delete `weight_halves` and the duplicated scale/zero, +keeping the existing single `F.linear`. The only reason that was not the +original design is that the tinygemm layout was believed opaque. It is not — +measured on this host (`torch._convert_weight_to_int4pack_for_cpu`, AVX-512): + +- The packed buffer is `[N/64][K][32 bytes]`: **K is the middle axis**, so a + K-range is contiguous *within each n-block of 64*. +- Reassembling a K-tile from those per-n-block segments is **byte-identical** to + packing that K-slice directly (verified exactly at N=2048, K=2048 for tile + widths 128/256/512), and split-K through tinygemm reconstructs the single-call + result at **51.5 dB**. +- The lane order inside a 32-byte run is a **fixed 64-entry interleave** — low + nibble = output channel *d*, high nibble = *d+32* — identical across every k + and every n-block. So decoding is a deinterleave, not a gather or a + reverse-engineered ISA table. + +**Do**: point the existing dequantizer at the tinygemm layout via that +deinterleave; drop `weight_halves`, `weight_group_scale`, `weight_group_zero`. + +**Expect**: most of the 740 MB back **at unchanged prefill**, because the GEMM +shape and reduction order do not change — only where the nibbles are read from. +The largest single temporary stays one dequantized `gate_up` (N=13312, K=2048, +bf16 = **52 MiB**), which already exists today. + +**Verify**: `RssAnon` (0.2%) plus pp512, three interleaved passes, and an SNR +assertion against the current dequant path — the check that caught the original +layout bug. + +**Risk**: tails. N not divisible by 64 or K odd need the scalar path; the +deinterleave rule above is verified only for full blocks. + +**Not doing first**: the draft's K-tiled dequant-and-accumulate. Even granting +the layout, calling oneDNN once per K-tile means the M×N accumulator is re-read +and re-written per tile — for `gate_up` at M=512 that is ~26 MiB of fp32 +accumulator × 15 extra tiles ≈ 780 MiB per layer per prefill, which is not the +"few MB of scratch" the draft claimed. Avoiding that traffic means keeping the +accumulator panel in registers across the K loop, i.e. writing a blocked GEMM +microkernel rather than calling a library per tile. If temporaries still need +cutting after the item above, **N-blocking is the better experiment** — disjoint +output columns need no cross-tile accumulation at all. + +### I2. Measure the native non-AMX 4-bit path before building anything + +`weight_path.py:14` calls tinygemm "the only option" without AMX. That is false: +when `layer.use_w4a8` is off, vLLM 0.28 falls through to `ops.cpu_gemm_wna16` +([cpu.py:202](../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py)), +a vector-ISA W4A16 kernel that has never been measured here. Both the ARM story +and the "fidelity" weight path rest on an untested assertion. + +**Do**: benchmark `cpu_gemm_wna16` against tinygemm at Spark's shapes, decode +and prefill. **Verify**: `bench_int4_kernels.py`. **Then** fix `weight_path.py`. + +This is a day's work that could invalidate a kernel project, so it comes first +among the kernel items. + +### I3. Asymmetric W4A8 — a fidelity experiment already supported by the kernel + +The report treats W4A8's zero point pinned at 8 as a kernel limitation. It is +not: `_process_gptq_weights_w4a8` uses the checkpoint's zero points whenever +`config.zero_points` is set, and synthesizes the constant only otherwise +([cpu.py:98](../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py)). +Our exporter chooses `"sym": true` +([quantize_gptq_int4.py:102](experiments/spark_edge/quantize_gptq_int4.py)). +W4A8 is the fastest path and the worst on task completion (1/12 greedy; the +agent navigates correctly then fails to stop). An asymmetric export is a +config change, not a kernel project. + +**Do**: re-export with `sym: false` and zero points, re-run fidelity and the +agent suite. **Expect**: unknown, but it is the cheapest fidelity experiment +available and it tests the *stated* cause of W4A8's error. + +### I4. A better quantizer — only after I3, and on an honest bit budget + +Ours reproduces 2/12 greedy prompts; llama.cpp Q4_K_M manages 4/12. The draft +called this "the same bit budget". It is not. Ours is **4.25 bits/weight** +(4 + two bf16 per 128-weight group; `weight_path.py` says 4.16, which is wrong); +plain Q4_K is **4.5 bits** over groups of 32, and Q4_K_M additionally promotes +some tensors to higher-bit types. Group size, per-tensor type selection and +rounding are all confounded, so "6-bit sub-block scales cause the gap" is an +attribution the experiment never isolated. + +**Do**: first close the budget gap cheaply (g64 and/or fp16 scale+zero, which is +a config change) and re-measure; only then implement error-compensated rounding +(GPTQ-style, calibration data exists) or AWQ-style scaling. + +**Verify**: greedy divergence *and* the scored agent suite — but only after H1 +below, because today's suite cannot rank builds. + +**Risk**: may produce nothing. Timebox it; a null result is worth recording. + +### I5. Attention op-chain fusion — worthwhile, but not "highest confidence" + +Four ops per layer, 112 dispatches, **1.3085 ms/step** measured +([profile_w4a16_g128_p2.json](experiments/spark_edge/profile_w4a16_g128_p2.json)): +`unified_attention_with_output` 0.4379, `unified_kv_cache_update` 0.3948, +`cpu_attention_with_kv_cache` 0.3471, `cpu_attn_reshape_and_cache` 0.1287. + +These are **two wrapper/kernel pairs**, not four independent kernels. Wrapper +self time is 0.833 ms — 64% of the total — so overhead does dominate at this +shape: at batch 1 the cache-update OpenMP loop runs over tokens × KV heads, i.e. +**two iterations**, while both paths independently rebuild cache views and the +wrapper allocates a dummy tensor for compile ordering. + +But the draft's supporting arithmetic was about a workload it never profiled. +"28 MB moved" assumes every layer attends 512 tokens; 7 of 28 layers are full +attention, and `profile_cpu_step.py` profiles a short prompt generating 64 +tokens — nowhere near a filled window. + +**Expect**: revised down. Eliminating *all* wrapper time saves 0.833 ms, and +some wrapper is required, so the draft's 0.6–0.9 ms demanded savings from the +C++ bodies too. Target **0.3–0.5 ms** (2–4% of step) from consolidating context +and cache-view handling through the existing `forward_includes_kv_cache_update` +interface, plus a serial path for the two-iteration cache update. + +**Verify**: per-op profile **and** unprofiled step latency — one outer call that +still launches two threaded kernels removes neither thread launch, and fewer +profiler events alone make this look better than it is. + +### I6. Import cost — an upper bound, not a saving + +`import vllm_omni` costs 19.6 s against `import vllm`'s 5.2 s; auto +image-processing (4.6 s) and the `openai` client (1.3 s) are identifiable inside +it. The draft promised 4–6 s per process. That 5.9 s is an **upper bound on +removable import work**, not a demonstrated reduction in time-to-ready: deferred +imports often just move the cost later. The init timer starts *after* imports +([bench_init_memory.py:44](experiments/spark_edge/bench_init_memory.py)), so the +current harness cannot measure this at all. And `patch.py` uses `ModelConfig` at +import time to install a descriptor patch +([patch.py:54](../vllm-omni/vllm_omni/patch.py)) — deferring it means changing +how that patch installs, not moving a statement. + +**Do**: first make the harness measure **process start → ready**; then defer. +**Verify**: that end-to-end number, with `-X importtime` as the diagnostic. + +### I7. A real memory ledger + +Prerequisite for any claim about a remainder. The draft's tools do not do the +job: `smaps_rollup` is an aggregate (use `smaps` per mapping), and `tracemalloc` +does not attribute native tensor storage or allocator arenas. + +**Do**: count unique tensor storages (both embedding layouts included), then +`smaps` by mapping for the rest. **Expect**: a decision about whether the +remainder is reducible — the draft's "unexplained 1.6 GB" was a subtraction from +a breakdown now known to be wrong. + +### I8. Resolve the `prefill_dequant: false` correctness hazard + +`cpu_int4.py:55` records that running tinygemm on a multi-row batch and then +`cpu_attention_with_kv_cache` in the same forward dies with SIGILL — an AMX tile +configuration bug — and sets `dequant_threshold: 2` to avoid it. But +`prefill_dequant: false`, the 2802 MB build the report calls the smallest +resident configuration, routes multi-row batches to exactly that path. Either +the 448 tok/s measurement and the crash report disagree about build or shape, or +that configuration is not deployable. **Do**: reconcile before quoting 2802 MB +as a build anyone can use. + +### I10. Megakernel / persistent thread pool -- the biggest measured headroom + +Raised after the review. On GPU a megakernel (Hazy Research's low-latency +Llama, Mirage's MPK) fuses a whole forward pass into one launch, removing +per-kernel launch cost and the pipeline bubble where the next layer's weights +cannot start loading. There are no kernel launches on CPU, so the analogues +are **OpenMP fork/join per op**, **ATen dispatch**, and the same cross-op +prefetch bubble. + +This is not a hypothetical for us: **llama.cpp already is one.** +`ggml_threadpool` starts its threads once and they walk the entire graph +together, synchronising at `ggml_barrier` through an atomic spin +(`ggml-cpu.c:490,576`). vLLM enters a separate `omp parallel` region inside +each kernel. So the comparison this whole study is built on is, in part, a +comparison of execution models. + +Measured here, so the prize is sized before anything is built: + +* **Parallel-region cost: ~1.2 us** at 24 threads (a just-above-grain-size op, + 24 threads vs 1). Spin-waiting does not help -- `OMP_WAIT_POLICY=ACTIVE` plus + `GOMP_SPINCOUNT=infinite` moved it 1.23 -> 1.18 us, inside noise -- so this + is barrier and scheduling cost, not thread wake-up, and the one-env-var + approximation of a persistent pool buys nothing. +* **Regions per decode step: a few hundred.** 113 quantized GEMMs and 112 + attention ops certainly parallelise; most elementwise tensors at batch 1 are + below PyTorch's grain size and run serially, so the 764 candidate calls in + the profile overstate it. At ~250-350 regions this is **0.3-0.4 ms, 2-3%** -- + real, but the same money as I5, which is far cheaper. +* **The bubble is the prize, not the barriers.** The GEMM sustains 60% of the + measured read ceiling and the whole step 33%. A persistent kernel that keeps + the weight stream running across op boundaries is the mechanism that could + close part of a **2.76 ms** gap. + +One of our own decisions works against this: `cpu_int4_linear` is deliberately +opaque to dynamo so the batch-size branch never becomes a guard on vLLM's +traced graph. That also stops inductor fusing anything across it. The guard +stability was worth it, but it is a cost worth naming. + +**Do**: *not* build a megakernel next. The honest read is that the fork/join +half is small and already covered by I5, and the streaming half needs the +execution model replaced -- a persistent pool plus cross-op pipelining inside +vLLM's CPU backend, fighting its attention backends, KV cache management and +torch.compile integration. That is weeks, and it reimplements ggml. + +What *is* worth doing, in order, because each one tests the same thesis for a +fraction of the cost: + +1. **Find out why the GEMM is at 60%.** It is not obviously bandwidth. The + tinygemm kernel dequantizes once per activation row, so part of that time is + arithmetic, and each of the 113 calls streams only a few MB -- too little for + the prefetcher to reach steady state. Attribute it before optimising it. +2. **Measure `cpu_gemm_wna16`** (I2) and W4A8's 69% against the same ceiling. + If one kernel is materially closer, the answer is a better kernel, not a + different execution model. +3. **I5**, which is the cheap slice of the fork/join half. + +**Verify**: per-op profile plus the streaming-bandwidth figure above, which is +now the denominator any kernel claim should be quoted against. + +### I9. Explicitly not planned + +- **More kernel fusion.** Four attempts, four losses: vLLM's fused CPU RMSNorm + is 3.6–4.5× faster in isolation and slower in situ (inductor already fuses the + native form with the residual add); fused `gelu_and_mul` 2.7× slower at + Spark's shape; `cpp.min_chunk_size` −6% decode/−19% prefill; an inline gate + reduction −44%. I5 is op-*count* reduction, a different mechanism. +- **MiniCPM-o CPU overlay.** Dropped. The 8 B thinker at bf16 is ~19 GB, so it + buys no edge deployability; and the draft's "three stages should beat the + two-stage 2.1×" is not how overlap works — the bound is ΣT/max(T), so stages + of 90/5/5 s give 1.11×. (That the measured two-stage ratio is 64.57/30.54 = + 2.114× when the bound for two tasks is lower means something else changed too, + which the init work should explain before anyone quotes it.) +- **Thread count and warm-up batch size as memory levers** — 64 MB and 2 MB. +- **KV-cache sizing is *not* on this list any more.** The 188 MB that retired it + was measured generating 8 tokens. At 32 k context `kv_budget.py`'s own formula + gives 1792 MiB flat vs 469 MiB hybrid — a 1323 MiB difference. It was measured + at a context where no cache is touched, which is not evidence that it never + matters. (`CpuPlatform.support_hybrid_kv_cache()` returns `True` in vLLM 0.28, + so the report's flat accounting needs a configuration explanation too.) +- **Speculative decoding** for tg128: it would inflate a fixed-prompt + `ignore_eos` benchmark without representing anything. Measure it on the agent + suite or not at all. + +--- + +## Harness + +The review's sharpest finding is that the harness cannot currently support the +comparisons the inference work needs. Correctness first, statistics second. + +### H1. Fix the scored suite before running it more + +- **The alarm oracle is wrong.** `add_alarm` passes if any typed string contains + `"7"` ([tasks.py:101](experiments/spark_edge/android_agent/tasks.py)) — not + 7:30, not AM, not that an alarm was saved. Typing `7` anywhere scores a pass. +- **The toggle task scores final state only**, so turning a setting off and back + on counts as success. +- **The scored suite is simulator-only**: task builders and predicates are typed + to `SimulatedDevice`. The real-device *episode* path exists; real-device + scoring and reset do not. +- **Repeats at temperature 0 over fixed fixtures measure repeatability, not + generalization** ([agent.py:48](experiments/spark_edge/android_agent/agent.py)). + +**Do**: correct the oracles; add paired task/layout/state variations so N runs +are N trials; implement real-device scoring and reset. + +This supersedes the draft's "N≥5 runs per task" — more repetitions of a wrong +oracle produce a confident wrong answer, which is the failure mode this project +already has three instances of. + +### H2. Fix measurement semantics + +- `run_grid`'s docstring says it rotates arms; it iterates `for pass: for arm in + arms`, i.e. the **same order every pass** + ([bench_harness.py:160](experiments/spark_edge/bench_harness.py)). It + interleaves but does not rotate, so position bias survives. +- `summarize` reports `best = max(values)` for every metric + ([bench_harness.py:123](experiments/spark_edge/bench_harness.py)) — wrong for + memory and init, where lower is better. Published "best" memory figures are + the worst pass. +- `profile_cpu_step.py` wraps all of `generate` (prefill + sampling) and divides + by `tg`, while the wall-clock path subtracts independent best-of-three + measurements. These are different quantities presented as one. +- Profile JSONs record no model path, context length or build; saved W4A8 and + W4A16 tags both read `bfloat16/None/eager=False`, and some saved W4A8 profiles + still contain the retracted bf16-head build with nothing marking them invalid. + +**Do**: rotate arm order per pass; take `best` by metric direction; separate +prefill from decode in the profiler; stamp provenance (model path, config hash, +commit, thread set) into every artifact and mark the retracted ones. + +### H3. Measure what the inference items need + +- **Process start → ready**, not engine-construction time (required by I6). +- **Peak `RssAnon` through load**, sampled, not the endpoint (required by I0). +- **Memory at realistic occupied context**, not 8 generated tokens (required to + re-open KV sizing). + +### H4. Contention guard + +Two sessions on the same cores wrecked several measurements and one grid's +control drifted 30% mid-run. A cooperative lock file only excludes cooperating +benchmarks, and load average > 8 identifies neither *which* cores are busy nor +whether the load is ours. + +**Do**: lock file plus a pre-flight check on per-core utilization for the +*target* core set, recorded in the artifact. + +### H5. Regression gates on measured bands + +The draft proposed an 8% throughput band. That 8% is a configured constant in +the harness, not an estimated interval, and it sits against acknowledged ~30% +run-to-run variation — it would fire constantly or never. + +**Do**: a checked-in expectations file whose bands come from the observed +distribution per metric (`RssAnon` ~0.2%, per-op self time ~5%, end-to-end +throughput wide enough to be meaningless, which is itself the argument for +gating on per-op time and memory instead). + +### H6. A denser fidelity metric + +Greedy divergence over 12 prompts saturates and cannot separate "slightly worse" +from "much worse". Top-1 agreement and KL against the bf16 reference over a few +thousand teacher-forced tokens, symmetric with `llama-perplexity +--kl-divergence` so llama.cpp stays comparable. Needs `llama-perplexity` built. + +Bring this forward: I3 and I4 are fidelity experiments and cannot be judged on a +saturated metric. + +### H7. Make profiling the default for small differences + +Per-op self time resolves ~5%; throughput on this host ~30%. Have +`bench_harness.py` say so when an arm difference falls inside the noise band, +and point at `profile_cpu_step.py`. + +### H8. Purge retracted numbers from executable source + +Retractions are still being shipped as documentation: +`kv_budget.py:9` advertises the withdrawn **8329 → 5254 MB**; +`weight_path.py` reports W4A16 as **4.16 bits** (it is 4.25), sets +`fused_cpu_norms=True` in three profiles against the recorded negative result +and the `"0"` default, and calls tinygemm "the only option" (see I2). + +--- + +## Order + +1. **H1, H2, H8** — the suite and harness cannot currently rank builds, and two + modules still assert retracted numbers. Everything downstream depends on this. +2. **H3** — add the three measurements the inference items are defined in terms + of (start→ready, peak RSS, memory at context). Then **H4**, **H5**, **H6**. +3. **I0** (peak load memory) — the actual edge blocker. +4. **I7** (memory ledger), **I6** (imports) — now measurable. +5. **I2** (measure `cpu_gemm_wna16`) — may invalidate a kernel project; cheap. +6. **I1** (single packed layout) — largest confirmed memory win at unchanged + prefill, and the layout facts it needs are now measured. +7. **I8** (SIGILL reconciliation), then **I3** (asymmetric W4A8) and **I4** + (quantizer) on a fixed fidelity metric. +8. **I5** (attention wrappers) — bounded overhead experiment, revised target. + +The draft ordered by "whose verification method already exists". The revision +orders by **whether the verification method is correct**, which is not the same +question and is the one this project keeps getting wrong. + +--- + +## Progress + +Started 2026-09-13, in the plan's own order. Items are struck through here only +when measured, not when coded. + +**Step 1 -- H1, H2, H8: done.** Commits `6424c69` (oracles and measurement +semantics), `27b94007` (retracted numbers out of `vllm-omni` source), `28aa63c`, +`b713d6e`, `4285f92`. Agent tests 27 -> 38, harness tests 9 -> 34, plus 7 for +the new fidelity metric. + +**Step 2 -- H3 to H7: done.** And it found a fifth measurement problem, of the +same kind as the first four: + +> **The memory benchmark was reading the wrong process.** It took +> `/proc/self/status`, which is the whole story only when the model is +> in-process. That happens when `VLLM_ENABLE_V1_MULTIPROCESSING=0` keeps the +> executor at `uni`; `vllm/platforms/cpu.py:273` promotes it to `mp` otherwise. +> Every earlier run in this study set that variable, so **every published +> memory figure is an in-process figure**. At vLLM's defaults the weights load +> into a worker the benchmark never looked at: a smoke run reported 796 MB +> resident for a model whose weights alone are 1.4 GB. Memory is now summed +> over the process tree. + +First numbers from the rebuilt benchmark (W4A16 g128, 24 threads, 2048 ctx, +`VLLM_CPU_KVCACHE_SPACE=1`, 2 interleaved passes): + +| | in-process (`uni`) | default (`mp`) | +|---|---|---| +| peak RssAnon, tree | 5722-5837 MB | 6068-6078 MB | +| after arena release | ~4480 MB | ~5544 MB | +| processes | 1 | 4 | +| `ready_s` (process start to serving) | 78.5-111 s | 88-153 s | +| of which imports | ~29-30 s | ~30 s | + +Three things follow. **The peak is 1.2-1.3 GB above the settled figure**, which +is I0's premise and is now measured rather than asserted. **vLLM's default +process layout costs ~240 MB of peak and ~1.1 GB of settled memory** over the +in-process one, which no published number reflected. And **`ready_s` is far +above `init_s`** -- imports are ~30 s, not the 19.6 s the plan quotes, so I6 is +worth more than it claimed and was previously unmeasurable. + +These are not comparable to the report's 3411 MB: that row was taken +in-process, with a sized KV budget, generating 8 tokens. Which is the argument +for the provenance stamping added in the same step. + +**Step 3 -- I0 (peak load memory): done.** The driver was in our own repack +path, not in vLLM: `unpack_nibbles` expands 4-bit codes to int32, and doing +that a whole matrix at a time makes the 131072-row tied embedding a **1024 MiB +transient**. Repacking in n-blocks (the layout makes them independent) is +byte-identical and cuts the load-time transient from 550 to 105 MB, **489 MB +off the peak**, with settled memory and init time unchanged -- chunked +repacking is in fact faster than whole-matrix (79 ms vs 208 ms for the +embedding). Chunk size tuned from a sweep rather than guessed. + +**Step 5 -- I2 (measure `cpu_gemm_wna16`): done, and it did not invalidate +anything.** The fallback is real, so `weight_path.py`'s "only option" was +false, but at 32.3 tok/s it is **2.2x slower than tinygemm** (71.6) and 2.3x +slower than W4A8 (74.4), all on the same cores with only the env var +differing. The routing was right; it now cites a measurement. Caveat recorded: +`_get_isa_hint` returns `"amx"` whenever AMX is present, so this measured the +kernel's best path on this host, not the `"vec"` path an AMX-less machine takes. + +**Step 6 -- I1 (single packed layout): done.** The layout +facts the item needed are now established on this host rather than assumed +(see I1 above), and the reconstruction is **bit-exact** against the existing +dequantizer at 2048x2048, 3072x2048 and 256x512 -- including `gate_up`'s real +13312x2048. `weight_halves` and the duplicated scale/zero are no longer +allocated where N tiles the 64-channel block, which every Spark linear does +(2048, 3072, 13312, 131072); shapes like N=80 keep the copy rather than +mis-read the layout. `VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT=0` restores the old +path, which is how the two are being A/B'd with identical code. + +*Process note, since this study keeps being bitten by measurement: I edited +`cpu_int4.py` while an unrelated grid was running, so that grid's later passes +used different code from its earlier ones. Those rows were discarded and the +comparison re-run under the escape hatch instead.* + +--- + +## Review record + +**Accepted and applied** (each verified in source before applying): the step +table conflated W4A8 and W4A16 (W4A16 is 12.51 ms, and the remainder is 1.60 ms +not ~1.9); bandwidth saturation of the GEMM does not make the rest overhead; +per-step savings were added when the measured total contradicts the sum; the +740 MB is a second *packed* copy plus duplicated scale/zero, not a bf16 matrix, +so the cheap fix is direct dequant rather than a tiled kernel; per-tile oneDNN +calls incur accumulator traffic the draft ignored; 4.25 ≠ 4.16 bits and Q4_K_M +is 4.5 bits with mixed types, so the fidelity comparison was never +budget-matched; the import figure is an upper bound and the init timer starts +after imports; the MiniCPM-o stage-count extrapolation is invalid and the item +does not serve edge deployability; peak load memory, not steady memory, is the +edge blocker; KV sizing was retired on an 8-token measurement; W4A8 supports +asymmetric zero points, so the report mistook our export choice for a kernel +limitation; `cpu_gemm_wna16` is a real non-AMX path and `weight_path.py`'s "only +option" is false; `smaps_rollup`/`tracemalloc` cannot deliver the attribution +I7 needs; `prefill_dequant: false` bypasses the documented SIGILL guard; the +harness does not rotate arm order, takes `max` as "best" for every metric, mixes +prefill into per-step timing and stamps no provenance; the alarm oracle accepts +a bare `"7"`, the toggle task scores final state only, and the scored suite is +simulator-only. + +**Corrected with new measurements.** The review held that the tinygemm layout +requires implementing "the ISA-specific lane mapping and tail handling" and that +a K interval "is not a plain column slice", concluding a tiled decoder is +expensive to get right. The first half is understated and the second is true but +not the obstacle it implies. Measured here: a K-tile is contiguous *within each +n-block of 64* and reassembling those segments is byte-identical to packing the +K-slice (N=2048, K=2048, tile widths 128/256/512), split-K reconstructs the +single call at 51.5 dB, and the lane order inside each 32-byte run is a fixed +interleave (low nibble → channel *d*, high → *d+32*) that is constant across +every k and n-block. So decoding needs a deinterleave, not a reverse-engineered +table — which is what makes I1's direct-dequant fix small enough to be the first +thing tried. The review's conclusion (do direct dequant, not the tiled kernel) +is unchanged and adopted; its reasoning about why is superseded. + +The review's accumulator-traffic objection assumes K-outer with a full M×N +accumulator, which a register-blocked GEMM would not pay — but since the draft +proposed calling a library per tile, the objection lands against the draft as +written, and I restructured rather than defended it. diff --git a/docs/analysis/spark_cpu_int4.md b/docs/analysis/spark_cpu_int4.md new file mode 100644 index 000000000..4d513d769 --- /dev/null +++ b/docs/analysis/spark_cpu_int4.md @@ -0,0 +1,237 @@ +# Closing the CPU decode gap: 4-bit weights for Spark-X2.5 on vLLM + +`analysis/spark_edge_support.md` ended with one open item. At matched bf16, +vLLM beat llama.cpp on Spark-X2.5-1.7B everywhere -- 8.7-15.4x on prefill, +1.19-2.65x on decode -- but llama.cpp's **Q4_K_M** build won single-stream +decode outright. Decode at batch 1 reads every weight to emit one token, so +bits per weight, not flops, set the token rate, and vLLM had no 4-bit CPU +path in use. This note closes that and re-runs the comparison at matched +precision. + +Two corrections to earlier findings come first, because both changed the +answer. + +Code: `analysis/experiments/spark_edge/{export_gptq_cpu,quantize_int4,ppl_vllm,profile_cpu_step,bench_vllm_w4a8,cmp_llamacpp,dequantize_int4}.py`, +`vllm_omni/model_executor/layers/quantization/cpu_int4.py` (+ tests). + +## 0a. Correction: the llama.cpp numbers were measured cold + +The Q4_K_M column in `spark_edge_support.md` was taken on a cold page cache. +The GGUF lives on a FUSE network filesystem and llama.cpp mmaps it, so the +first grid was partly timing storage. Warm, same pinned cores: + +| llama.cpp Q4_K_M, 32 threads | pp512 | tg128 | +|---|---|---| +| as previously published (cold) | 1199 | 82.81 | +| warm | **1356** | **98.02** | + +The bar this work has to clear is 98 tok/s, not 82.8. + +## 0b. Correction: vLLM's CPU backend *does* ship a 4-bit kernel + +`spark_edge_support.md` reported that vLLM 0.28's CPU build has no int4 dense +kernel, on the evidence that `woq_int4_linear`, `int4_scaled_mm_with_quant` +and `da8w4_linear` are all absent. They are -- but the kernel exists under a +different name. `torch.ops._C.int4_scaled_mm_cpu` is present in the released +CPU wheel: an **AMX W4A8** GEMM, reached by loading a GPTQ checkpoint with +`VLLM_CPU_INT4_W4A8=1`, with `cpu_gemm_wna16` as the W4A16 fallback. The +search missed it because it searched for names, not for the capability. + +This matters, because it is the fastest option by a wide margin. Measured at +Spark's shapes against PyTorch's `aten::_weight_int4pack_mm_for_cpu` +(`bench_vllm_w4a8.py`, 28 layers + head, group 128): + +| batch | tinygemm | vLLM W4A8 (AMX) | speedup | +|---|---|---|---| +| 1 row (decode) | 4.45 ms | **2.66 ms** | 1.67x | +| 8 rows | 14.25 ms | **6.76 ms** | 2.11x | +| 512 rows (prefill) | 878 ms | **104 ms** | 8.44x | + +## 1. What was actually slow + +`profile_cpu_step.py` attributes one decode step. At bf16, 32 threads: + +| bucket | ms/step | share | +|---|---|---| +| `_C::onednn_mm` (113 GEMMs) | 21.24 | 76% | +| inductor-generated kernels (norms, rope, GEGLU) | 2.47 | 9% | +| attention + KV cache ops (4 x 28) | 1.63 | 6% | +| head-gate `aten::mm` (28) | 1.08 | 4% | +| sampling (`argmax` over 131072) | 0.24 | 1% | +| ~900 small view/slice ops | 0.55 | 2% | +| **total** | **27.96** | 35.8 tok/s | + +3.42 GB of bf16 weights at ~161 GB/s effective. Only fewer bits move it. + +## 2. Results + +Xeon Platinum 8480C, one socket, 32 physical cores pinned with `taskset`, +warm page cache, engines run strictly one at a time, and every run records the +busy fraction of its own cores before it starts (`cpu_busy.py`) -- a +shared-core benchmark measures the scheduler, not the engine. Perplexity is +teacher-forced over 512-token windows, 126 217 scored tokens, each engine +scored against **its own** bf16 run so tokenizer and chunking differences +cancel. + +| build | bits/weight | pp512 | tg128 | ppl vs own bf16 | +|---|---|---|---|---| +| vLLM bf16 | 16 | 4455 | 36.35 | -- | +| vLLM GPTQ W4A16 (sym, g128) | 4.16 | **4659** | 37.53 | +25.5% | +| **vLLM GPTQ W4A8 (sym, g128)** | 4.16 | **4355** | **88.7** | +27.2% | +| vLLM `cpu_int4` (asym, g128) | 4.25 | 3589 | 85.5 | +18.2% | +| vLLM `cpu_int4` (asym, g64) | 4.50 | 3689 | 81.1 | +12.5% | +| llama.cpp Q4_K_M | 5.15 | 1356 | **98.02** | **+7.6%** | + +Against llama.cpp at 4-bit: **prefill 3.21x faster, decode 0.90x.** Decode +went 36.35 -> 88.7, a 2.44x gain, which turns a 2.70x deficit into a 1.11x +one without quite closing it. + +Quantizing the head matters more than anything else in that table: Spark ties +its 131072 x 2048 vocab table to the output head, and that one GEMM is the +largest read in a decode step. Leaving it in bf16 costs 12 tok/s (76.0 vs +88.7), so `export_gptq_cpu.py --quantize-lm-head` unties it, keeping the +table in bf16 for the lookup (a few hundred bytes a token) and giving the +head its own 4-bit copy. + +Note the W4A16 row: best prefill of anything measured, worst decode of the +4-bit builds. It dequantizes to bf16 per call, which amortizes over 512 rows +and is pure overhead on one. + +### Thread scaling (cores 56-87, both engines, default OpenMP settings) + +Measured with `cpu_int4`, before the W4A8 path was found, but the shape of it +is the point: + +| threads | vLLM int4 pp512 | llama.cpp pp512 | vLLM int4 tg128 | llama.cpp tg128 | +|---|---|---|---|---| +| 4 | **800.1** | 210.2 | **27.08** | 24.20 | +| 8 | **1476.2** | 409.8 | 44.02 | 44.00 | +| 16 | **2599.1** | 799.2 | 66.72 | **73.19** | +| 32 | **3524.9** | 1348.0 | 80.20 | **99.55** | + +Decode wins at 4 threads, ties at 8, loses from 16 up. That is the signature +of a fixed per-step cost that does not shrink when the GEMMs do. + +### What else moved the needle, and what didn't + +| change | tg128 | | +|---|---|---| +| `cpu_int4` g64 baseline | 81.10 | | +| `KMP_BLOCKTIME=infinite OMP_WAIT_POLICY=ACTIVE KMP_LIBRARY=turnaround` | 83.61 | ~250 OpenMP parallel regions per step; how threads park between them is not free | +| group 128 instead of 64 | 85.48 | 5.5% fewer weight bytes | +| `detokenize=False` (what `llama-bench` measures) | 82.13 | | + +Tried and rejected: `compile_sizes=[1]`, to give inductor static shapes for +the decode step (81.77, no change); `enforce_eager` (58.88, much worse); +inductor `cpp_wrapper` (fails outright -- vLLM's own ops take a `str` +argument `custom_op_wrapper` cannot box). + +## 3. The remaining decode gap is framework, not kernel + +Re-profiling the `cpu_int4` step at 12.50 ms: + +| bucket | ms/step | +|---|---| +| int4 GEMMs (113) | 5.49 | +| inductor-generated kernels | 2.29 | +| attention + KV cache ops | 1.41 | +| head-gate `aten::mm` (28) | 0.92 | +| custom-op dispatch (113) | 0.58 | +| sampling + small ops | 0.80 | +| **total** | **12.50** | + +Those GEMMs move 0.95 GB in 5.49 ms, **173 GB/s**; llama.cpp does its whole +10.2 ms step against 1.1 GB, ~108 GB/s all-in. The 4-bit kernel is not what +is losing. **7.0 ms of the 12.5 ms step is per-step framework cost**, and +llama.cpp has almost none of it. With the faster W4A8 kernel the GEMM share +falls further and the framework share rises, which is why the gap closes to +0.90x and then stops. + +Closing the rest is vLLM CPU engine work -- the attention wrappers, the ~900 +small view ops, inductor launch overhead -- and it is out of reach here +because vLLM itself is off-limits to modification. The one piece inside reach +is the head gate: 28 `aten::mm` calls, 0.92 ms, almost all thread barrier for +a 2048x8 matmul. Rewriting it as a fused broadcast-and-reduce was tried and is +much worse (44.5 tok/s; inductor does not vectorize the reduction). Folding +those eight rows into the fused QKV GEMM would remove it, and is the obvious +next step. + +## 4. Quality: the grid, not the runtime + +Q4_K_M spends **5.15 bits per weight** (1.1 GB for 1.7 B parameters) against +4.16-4.50 here, and still loses less: +7.6% perplexity against +12.5% for the +best of ours. Two separate things cause that. + +**vLLM's GPTQ path cannot express an asymmetric grid at 4 bits.** It rejects +`sym=False` outright ("Unsupported quantization config: bits=4, sym=False"), +so the fast W4A8 kernel can only be fed a symmetric grid -- which wastes a +level on every group whose weights are off-centre. That is most of why the +W4A8 rows cost +27% while `cpu_int4`, which is asymmetric, costs +12.5% at a +comparable bit budget. The AMX kernel itself takes real zero points +(`convert_weight_packed_scale_zp(..., CPUQuantAlgo.AWQ)`); it is the GPTQ +*config* that refuses. Exporting AWQ format instead should get an asymmetric +grid through the same kernel, and is the highest-value thing left undone +here. + +**W4A8 quantizes activations to int8, and that is nearly free.** W4A16 on the +same checkpoint scores +25.5% against W4A8's +27.2% -- a 1.7 point difference +for a 2.4x decode speedup. That is a much better outcome than int8 +activations have had elsewhere in this repo (Qwen3-TTS, MiniCPM-o, Spark on +the NPU), and worth recording as the counter-example. + +**Even asymmetric, our grid is coarser than K-quants.** `cpu_int4` does +round-to-nearest on each group's min/max with a small shrink search; the +search is worth about what it costs (at g64: 48.0 perplexity with it, 59.7 +without, on the smaller corpus) but it optimizes per-group squared error, +which is the wrong objective. Q4_K_M uses an importance-weighted scale search +per 32-weight sub-block and spends more bits on the tensors that need them. + +This is a quantizer gap, not a runtime gap, and the control proves it: +expanding a quantized checkpoint back to plain bf16 offline +(`dequantize_int4.py`) and scoring that reproduces the quantized run to +**within 0.2%** (48.08 vs 48.01 at g64, 56.80 vs 56.89 at g32). Both GEMM +paths, the nibble packing and the quantized tied embedding are all faithful. + +A caution on measuring this: on a smaller, harder corpus (34 k words of this +repo's own notes, bf16 perplexity 43.7) the ordering is not even stable -- +group 32 lands worse than group 128 there, and reverses on the larger corpus. +Perplexity differences between 4-bit variants of a 1.7 B model need a corpus +big enough to be in a sane regime before they mean anything. + +## 5. An AMX hazard worth recording + +Running PyTorch's tinygemm kernel on a multi-row batch and then vLLM's +`cpu_attention_with_kv_cache` in the same forward kills the process with +**SIGILL inside the attention kernel** -- the signature of an AMX instruction +issued with no valid tile configuration loaded. The identical prompt is fine +in bf16, and fine at any batch size once every multi-row GEMM takes a +different path, so the two kernels disagree about who owns the AMX tile state. +`cpu_int4` therefore routes only single-row steps to tinygemm +(`dequant_threshold` defaults to 2). vLLM's own W4A8 path does not have the +problem. + +## 6. Where this leaves it + +| | before | after | +|---|---|---| +| decode vs llama.cpp Q4_K_M | 0.37x (36.35 vs 98.02) | **0.90x** (88.7) | +| prefill vs llama.cpp Q4_K_M | no 4-bit path in use | **3.21x** (4355 vs 1356) | +| quality cost of 4-bit | -- | +27.2% ppl (W4A8) / +12.5% (`cpu_int4` g64) vs Q4_K_M's +7.6% | +| model on disk | 3.4 GB | 0.9-1.4 GB | + +Honest summary: 4-bit weights turn a 2.7x single-stream decode deficit into a +1.11x one and put prefill 3.2x ahead, but vLLM still does not beat llama.cpp +at single-stream decode on 32 cores, and our 4-bit grids are measurably +coarser than K-quants. Three specific, measured things stand between here and +a win, in order of expected value: + +1. **An AWQ-format export**, to get an asymmetric grid through the AMX W4A8 + kernel -- worth roughly the +27.2% -> +12.5% perplexity difference at no + speed cost. +2. **Folding the head gate into the QKV GEMM** -- 0.92 ms of a ~11 ms step. +3. **vLLM CPU per-step overhead** -- 7.0 ms at batch 1, the whole remaining + gap, and not addressable without changing vLLM. + +`cpu_int4` keeps a narrower role than intended: it is the more accurate of the +two (asymmetric grid, +12.5% against +27.2%) and it needs no AMX, but on an +AMX host vLLM's own W4A8 path is faster on both axes. Its docstring says so. diff --git a/docs/analysis/spark_edge_support.md b/docs/analysis/spark_edge_support.md new file mode 100644 index 000000000..d75ac8c88 --- /dev/null +++ b/docs/analysis/spark_edge_support.md @@ -0,0 +1,994 @@ +# Spark-X2.5-1.7B on CPU and Galaxy S25 + +Spark-X2.5 (XHToken/iFlytek, Apache 2.0, released 2026-09-01) is a 1.7 B +on-device model built for agentic tool use: BFCL-V4 46.9, tau-squared-bench +65.3, a native 1 M-token context and 200+ languages. This note covers adding +it to vLLM-Omni, measuring it against every engine that can run it, putting it +on a real Snapdragon 8 Elite, and driving an Android phone with it. + +Reproduction scripts: `analysis/experiments/spark_edge/`. +Model code: `vllm_omni/model_executor/models/spark2_5/`, exporter +`vllm_omni/edge/spark_export.py`. + +## 1. What had to be built + +Three traits separate Spark from a Llama-style decoder, and all three are why +no generic converter picks it up: + +| trait | detail | why it matters | +|---|---|---| +| hybrid attention | `layer_types` = 3x `sliding_attention` (window 512) then 1x `full_attention`, 28 layers -> 21 sliding + 7 full | bounds KV growth; this is what makes 1 M context affordable on a phone | +| per-layer-type RoPE | sliding: all 256 head dims, theta 1e4. full: first 64 dims only (`partial_rotary_factor` 0.25), theta 5e6 | two rotary tables, not one | +| head-wise output gate | `g_proj` [2048 -> 8] reads the same post-norm input as QKV; `sigmoid` scales each head's attention output before `out_proj` | an extra projection with no analogue in Llama/Qwen | + +Also: QKV ships pre-fused as one `q_k_v_proj`, the MLP is GEGLU with *exact* +gelu, embeddings are tied, and `head_dim` is 256 with only 2 KV heads. + +### Engine support before this work + +| engine | Spark-X2.5 support | +|---|---| +| llama.cpp | yes, native (`src/models/spark2-5.cpp`) | +| SGLang | yes, native (`srt/models/spark2_5.py`) | +| vLLM 0.28 | **no** -- added here | +| MNN | no. `llmexport.py` dispatches on a fixed `model_type` list; `spark2_5` is absent | +| ExecuTorch | no | +| ncnn | no | + +MNN, ExecuTorch and ncnn would each need the architecture written from +scratch. That is the honest reason they do not appear in the benchmark tables +below: they cannot load the model at all, not that they are slow at it. + +### Correctness + +Verified against the official remote-code implementation pinned to +transformers 4.57.1 (the version `config.json` targets; it does not import on +the 5.x in our venvs, so a dedicated venv runs ground truth). + +| scenario | prompt | fp32 | bf16 | +|---|---|---|---| +| short | 28 tok | 32/32 greedy, max logprob diff 2.2e-05 | 24/24 greedy | +| long (exercises sliding window) | 2465 tok | -- | 32/32 greedy | +| tool calling | 137 tok | 32/32 greedy | diverges at token 23 | + +The bf16 tool-calling divergence is a near-tie under greedy decoding, not a +defect: in fp32 the same prompt matches exactly with a 2.2e-05 logprob +difference. + +One vLLM bug had to be worked around. The head-wise gate is a [2048, 8] +projection, and vLLM's CPU backend routes any narrow linear to the sgl-kernel +packed GEMM, whose fake-tensor rule assumes the packed weight keeps N in dim +0. VNNI packing does not for N that small, so it reports K instead of N and +`torch.compile` dies tracing the gate multiply. The gate is kept as a plain +parameter with `F.linear` -- a 2048x8 matmul gains nothing from the packed AMX +kernel anyway. + +## 2. CPU: vLLM-Omni vs llama.cpp + +Xeon Platinum 8480C (Sapphire Rapids, AMX-BF16/INT8). Both engines pinned with +`taskset` to the same physical cores and run **strictly one at a time**; +prefix caching disabled on vLLM so a replayed prompt is really recomputed +(llama-bench has no such cache). pp512 = prefill throughput, tg128 = decode. + +### Same precision (bf16) + +| threads | vLLM pp512 | llama.cpp pp512 | prefill | vLLM tg128 | llama.cpp tg128 | decode | +|---|---|---|---|---|---|---| +| 4 | **1113.4** | 72.4 | 15.4x | **11.11** | 4.20 | 2.65x | +| 8 | **2113.7** | 173.8 | 12.2x | **19.88** | 16.15 | 1.23x | +| 16 | **3639.4** | 341.7 | 10.7x | **30.16** | 25.31 | 1.19x | +| 32 | **4454.9** | 509.5 | 8.7x | **36.35** | 28.57 | 1.27x | + +vLLM wins both, at every thread count. The prefill gap is AMX: llama.cpp's +bf16 path runs at AVX-512 speed, vLLM's goes through oneDNN/AMX tiles. + +### Against llama.cpp's quantized builds + +| threads | vLLM bf16 pp | llama.cpp Q8_0 pp | llama.cpp Q4_K_M pp | vLLM bf16 tg | llama.cpp Q8_0 tg | llama.cpp Q4_K_M tg | +|---|---|---|---|---|---|---| +| 4 | **1113.4** | 118.1 | 103.6 | 11.11 | 9.85 | **10.36** | +| 8 | **2113.7** | 490.2 | 434.2 | 19.88 | 45.01 | **42.40** | +| 16 | **3639.4** | 914.7 | 806.3 | 30.16 | 66.23 | **71.50** | +| 32 | **4454.9** | 1493.1 | 1199.0 | 36.35 | 76.54 | **82.81** | + +Prefill: vLLM at bf16 still beats llama.cpp's *best quantized* build by 3.0x +at 32 threads. Prefill is compute-bound, and AMX beats 4-bit weight savings. + +Decode: llama.cpp Q4_K_M wins from 8 threads up, by 2.3x at 32. Decode is +memory-bound -- 4-bit weights move a quarter of the bytes -- and this is a +precision difference, not an engine difference. Closing it needs a quantized +vLLM CPU path for Spark, which is the open item in section 6. + +### Quantized: int8 on CPU + +vLLM 0.28's CPU build has int8 dense kernels (`int8_scaled_mm_with_quant`, +`dynamic_scaled_int8_quant`), and `static_scaled_fp8_quant` is registered with +an empty schema, so on-the-fly fp8 fails outright. + +An earlier draft of this note said the backend had no dense int4 kernel +either, on the strength of `woq_int4_linear`, `int4_scaled_mm_with_quant` and +`da8w4_linear` all being absent. That was wrong -- they are simply not the +names it uses. See section 2c: there are two four-bit paths, and the one +vLLM ships is `int4_scaled_mm_cpu`, an AMX W4A8 GEMM reached through the +GPTQ/AWQ kernel selector. + +Int8 is also available. `analysis/experiments/spark_edge/quantize_int8.py` writes a +compressed-tensors W8A8 checkpoint straight from the safetensors (symmetric +per-output-channel round-to-nearest; no calibration needed, and llmcompressor +is avoided because it pulls transformers 5.x, on which Spark's remote code +does not import). The head-wise gate is left in full precision: its eight +outputs each scale a whole attention head, and it is 0.001% of the weights. +Weight SNR 36.9-40.9 dB across 140 tensors. + +| threads | pp512 bf16 -> int8 | tg128 bf16 -> int8 | +|---|---|---| +| 16 | 3639 -> **5135** (1.41x) | 30.16 -> **48.30** (1.60x) | +| 32 | 4455 -> **6978** (1.57x) | 36.35 -> **58.44** (1.61x) | + +Quality against the fp32 reference, though, is not free: + +| scenario | greedy match | first-token top-1 | top-20 overlap | max logprob diff | +|---|---|---|---|---| +| short (28 tok) | 23/24 | agrees | 18/20 | 0.83 | +| long (2465 tok) | **12/32** | agrees | 19/20 | 1.56 | +| tool calling | 31/32 | agrees | 18/20 | 0.58 | + +bf16 matched 32/32 on all three with a max difference of 0.24. Int8 keeps the +first token everywhere, but at long context the distribution has moved enough +that greedy output diverges after a dozen tokens. Usable, not free -- the same +caution the Qwen3-TTS and MiniCPM-o work in this repo reached about automatic +int8. + +## 2b. CPU under concurrency + +Single-stream decode is memory-bound, which is a precision contest llama.cpp +wins with 4-bit weights. Concurrent decode is compute-bound, which is an AMX +and continuous-batching contest. Total generated tokens/s, 32 pinned cores, +128-token prompts generating 128 tokens each: + +| batch | vLLM bf16 | vLLM int8 | llama.cpp Q4_K_M | int8 vs llama.cpp | +|---|---|---|---|---| +| 1 | 34.6 | 54.9 | **91.1** | 0.60x | +| 2 | 69.9 | 105.2 | **129.6** | 0.81x | +| 4 | 139.8 | 210.3 | **251.6** | 0.84x | +| 8 | 266.1 | 387.8 | **406.5** | 0.95x | +| 16 | 495.7 | **694.7** | 613.9 | 1.13x | +| 32 | -- | **1080.6** | 778.3 | 1.39x | +| 64 | -- | **1557.9** | 904.5 | 1.72x | + +The crossover is around batch 12. Below it llama.cpp's 4-bit weights win; +above it vLLM's AMX-INT8 GEMMs and continuous batching pull away, reaching +1.72x at batch 64 and still climbing. A phone agent is a batch-of-one +workload, so llama.cpp is the right single-device choice today; an edge +gateway serving a dozen sessions is vLLM's. + +## 2c. Four bits on CPU: two paths, and which one a device should take + +vLLM 0.28's CPU build has a 4-bit GEMM. `CPUWNA16LinearKernel` +(`vllm/model_executor/kernels/linear/mixed_precision/cpu.py`) routes GPTQ and +AWQ checkpoints to **`int4_scaled_mm_cpu`**, an AMX W4A8 kernel, whenever the +activations are bf16, the ISA has AMX tiles and the checkpoint carries no +activation reordering. Four-bit weights times eight-bit activations is exactly +what llama.cpp's Q4_K_M does (Q4_K weights, Q8_K activations), so this is the +honest precision match rather than a handicap. + +Two paths were built, because they are not the same trade: + +| | weights | activations | kernel | bits/weight | reached by | +|---|---|---|---|---|---| +| **W4A8** | 4-bit symmetric, group-wise | int8 (per the kernel) | `_C::int4_scaled_mm_cpu` (AMX) | 4.16 @ g128 | GPTQ checkpoint, vLLM's own dispatch | +| **W4A16** | 4-bit asymmetric min/max, group-wise | bf16 | `aten::_weight_int4pack_mm_for_cpu` | 4.16 @ g128 | `cpu_int4`, registered out-of-tree | + +W4A8 is forced symmetric: `_process_gptq_weights_w4a8` builds its own zero +point (a constant 8) and ignores the checkpoint's `qzeros`, so the grid must be +`w = (code - 8) * scale`. W4A16 is free to use an asymmetric min/max grid, and +a per-group scale search recovers roughly a dB in either case. + +### Which is faster + +Per-op profiles, taken back to back on the same 24 cores. Throughput on this +host swings ~30% run to run and the gap here is ~8%, so the GEMM self time is +the measurement that can actually rank them: + +| path | GEMM/step | step | tok/s | +|---|---|---|---| +| W4A8 g128 | **5.91 ms** | **11.55 ms** | **86.6** | +| W4A16 g128 | 6.85 ms | 12.51 ms | 80.0 | +| llama.cpp Q4_K_M | -- | 11.11 ms | 90.0 | +| vLLM bf16 | -- | -- | 41.0 | + +W4A8 wins twice: the kernel is 0.94 ms cheaper per step, and because it is +reached through vLLM's own dispatch it also avoids the 0.48 ms/step custom-op +boundary the tinygemm path needs (that boundary exists because tinygemm's cost +is linear in batch size, so prefill has to be routed to a dequantize-then-AMX +path instead -- see below). + +**The trap that cost a day: a GPTQ checkpoint must quantize the output head.** +Spark ties its embedding to its head, so a converter that leaves the tie in +place emits no `lm_head` tensors and that one GEMM runs bf16 through +`_C::onednn_mm` -- **2.97 ms per step**, more than the entire 4-bit saving. The +profile is what caught it: `int4_scaled_mm_cpu` ran 112 times (28 layers x 4 +linears) and `onednn_mm` once. Every "W4A8" number measured before that was on +a handicapped build, which is why the W4A8/W4A16 ordering appeared to flip +between grids. + +### Which is more faithful + +Greedy divergence from each build's own bf16 output, 12 prompts, measured the +same way on both engines: + +| build | exact match | mean prefix kept | +|---|---|---| +| llama.cpp Q4_K_M | **4/12** | **0.51** | +| vLLM W4A16 | 2/12 | 0.30 | +| vLLM W4A8 | 1/12 | 0.13 | + +Q4_K_M's 6-bit sub-block scales buy real granularity at the same bit budget; +round-to-nearest with a per-group search does not match it. So the two paths +are a genuine choice, not a ranking: **W4A8 for speed, W4A16 for fidelity**, +and `vllm_omni/edge/weight_path.py` makes that choice from a hardware probe +rather than a flag, because the fast kernel on each device needs a different +and mutually exclusive weight layout: + +| device class | path | why | +|---|---|---| +| x86 + AMX | W4A8 / GPTQ | `int4_scaled_mm_cpu` applies | +| x86 without AMX | W4A16 / tinygemm | no AMX, no int4 kernel | +| x86 + AMX, fidelity asked | W4A16 / tinygemm | keeps bf16 activations | +| ARM64 | W4A16, **unverified** | PyTorch's 4-bit CPU GEMM dispatches a different kernel there | +| phone NPU | exported graph, calibrated w8a16 | vLLM does not run there at all | + +### Prefill needs the other kernel + +The tinygemm kernel dequantizes once per row of activations, so its cost is +linear in batch size: it beats AMX bf16 below ~8 rows and loses badly above. +Measured at Spark's `gate_up` shape, per call: 0.070 ms at 1 token, 14.2 ms at +512. Left alone, pp512 collapsed from 4455 tok/s (bf16) to 557. Unpacking the +weight to bf16 once per call and handing it to the ordinary oneDNN/AMX GEMM +brings it back to **3347 tok/s**, 3.0x llama.cpp's 1121. W4A8 needs none of +this -- the AMX kernel is the right one at every batch size, which is the +second reason it is the default on an AMX machine. + +## 2d. Init time and resident memory, which is what decides edge at all + +Speed is not the binding constraint on an edge box; a 90-second start and 8 GB +resident is. Both turned out to be fixable, and one of them was a bug. + +### vLLM recompiled the model on every single start + +`_try_load_aot_compiled_fn` finishes a cache load with +`loaded_fn._artifacts.compiled_fn.finalize_loading(cfg)`. On CUDA that artifact +is a wrapper object carrying the method; with the inductor CPU backend it is a +plain function, so the call raises `AttributeError`, the handler treats the +whole load as a cache miss ("Compiling model again due to a load failure"), and +the model is compiled again from scratch. Every restart paid it. + +Attaching the missing no-op (a plain Python function accepts attribute +assignment; the CPU artifact loads its generated `.so` lazily on first call, +which is the warm-up run) fixes it. Same cache directory, one variable changed: + +| | init | +|---|---| +| cold cache (first ever compile) | 93.8 s | +| warm cache, vLLM as shipped | 49.8 s | +| **warm cache, shim installed** | **14.9 s** | + +**3.3x faster warm start**, reproduced across passes (14.9 / 14.7 / 15.0, and +12.8-13.0 on the W4A16 path). The shim is self-disabling: if the artifact +already has `finalize_loading`, as it does on GPU and will if upstream gives +the CPU path one, it changes nothing. + +### Measure anonymous memory, not RSS + +A first pass concluded that sizing the KV cache saved 3.1 GB. It does not, and +the error is worth recording because it is easy to repeat: that comparison +read `VmRSS`, which counts the mmap'd checkpoint's file pages, and those come +and go with page-cache pressure. **Identical configurations measured 1.2-2.0 GB +apart.** `RssAnon` -- allocations only -- repeats to within **0.5 MB**, and is +what a memory budget is actually about. + +### The KV cache is not a memory lever on CPU + +Re-measured properly, with only the budget changing: + +| KV budget | anonymous RSS | +|---|---| +| 4 GiB (env default) | 7601 MB | +| 512 MiB | 7405 MB | +| 128 MiB | 7413 MB | + +Shrinking the cache 32x moves resident memory by **188 MB**, and 512 MiB is +indistinguishable from 128 MiB. The cache is allocated but its pages are never +touched, so it costs address space, not memory. + +It still has to be *correct*, which is a separate problem: `--kv-cache-memory-bytes` +was silently discarded on CPU. `CpuPlatform.check_and_update_config` ends with +a block its own comment calls "Lagecy setting" that overwrites +`cache_config.kv_cache_memory_bytes` from `VLLM_CPU_KVCACHE_SPACE` +unconditionally, so the budget could only be set in whole GiB and vllm-omni's +per-stage budgets had no effect at all. Verified by allocation rather than by +memory, because allocation cannot drift: with the env at 1 GiB, asking for +128 MiB produced the same **17 881** cache tokens as asking for nothing, and +with the env at 4 GiB it produced **71 527**. After restoring the obvious +precedence the same request yields **2 231** tokens. That matters for +admission -- vLLM preempts by recompute when the cache cannot hold +`max_num_seqs x max_model_len`, which on a single-stream deployment is a stall +-- not for footprint. `vllm_omni/edge/kv_budget.py` prices the cache from the +config, reporting the flat figure (what is safe to pass, since vLLM's CPU +accounting is flat) beside the hybrid one a layer-type-aware backend could use. + +### Where the memory actually goes: the weight load + +Split by phase, on anonymous RSS: + +| phase | W4A8 (GPTQ) | W4A16 (tinygemm) | +|---|---|---| +| before import | 12 MB | 12 MB | +| after `import vllm_omni` | 513 MB | 515 MB | +| after engine construction | 7338 MB | 5023 MB | +| **after releasing free arenas** | **3510 MB** | **3411 MB** | + +Imports cost half a gigabyte -- they are the init-*time* problem, not the +memory one. The engine adds 4.5-6.8 GB for a model whose weights are under +1.4 GB, and **most of it is not live**: loading a quantized checkpoint unpacks +4-bit codes to int32 -- four bytes per weight -- to permute and repack them, +and although the intermediates are freed the allocator keeps the arenas. One +release call returns **3828 MB** on the GPTQ path and 1612 MB on tinygemm, a +**52%** cut on the configuration we actually want to deploy. + +The second reading of that table matters more than the first: after the +release both paths land in the same place. W4A8's apparent 2.3 GB penalty over +W4A16 was never a footprint difference, it was the repack. So W4A8 wins on +speed *and* ties on memory, and fidelity is the only remaining reason to +choose W4A16. + +`vllm_omni/edge/memory.py` does the release and `patch.py` hooks it to the CPU +worker's load and warm-up (`VLLM_OMNI_CPU_RELEASE_MEMORY=0` to disable). The +~3.5 GB that remains, against 0.86 GB of weights, is the next thing to chase. + +### Two bugs that made CPU deployments take the slow path silently + +Init is not one number per model; it has a different cause in each pipeline, +and in two places the slow path was taken without saying so. + +**Parallel stage init could never run on CPU.** +`resolve_stage_physical_devices` returns `None` for `devices: "cpu"`, and the +admission resolver fails closed on `None` -- right for a GPU replica whose +devices cannot be accounted, wrong here -- so every multi-stage CPU pipeline +raised `StageAdmissionError` and fell back to sequential startup. The edge +profile *sets* `parallel_stage_init: true`, so on CPU that profile was broken +rather than merely slower. + +The fix is not to skip admission. The per-device ledger is vacuous on CPU, but +the resource concurrent init contends for is host memory, and two stages +allocating KV caches at once exhaust it exactly as two GPU stages exhaust a +card. `check_host_admission` applies the same discipline to that resource: sum +the declared budgets, compare with available RAM and a margin, refuse to start +otherwise, and fail closed on a stage that declares no budget. + +| Qwen3-TTS 0.6B, init only | | +|---|---| +| sequential | 64.57 s | +| **parallel** | **30.54 s** | + +**A deploy YAML meant different things depending on which flag loaded it.** +The `orchestrator:` block carrying `parallel_stage_init` was applied only on +the `deploy_profile` path, so pointing `--deploy-config` at the very same file +loaded its stages and dropped the flag: 63.76 s against 30.76 s, with nothing +in the log to explain the difference. The block now applies either way. + +### What is not a memory lever + +Two candidates tested and closed, so they are not re-tried: + +| lever | effect | +|---|---| +| OMP thread count (8 / 24 / 48) | 3754 / 3775 / 3818 MB -- 64 MB across a 6x range | + +**Retracted (2026-09-14): KV cache size was on this list, and it is the single +largest lever there is.** The row read "4 GiB -> 128 MiB saves 188 MB, the +pages are never touched". Re-measured now that the budget is actually +honoured -- 3 interleaved passes per arm, both filling a 1536-token context: + +| KV budget | resident | +|---|---| +| 1 GiB (the smallest `VLLM_CPU_KVCACHE_SPACE` can express) | 3264 MB | +| 192 MiB, explicit | **2421 MB** | + +**843 MB.** And `mincore(2)` says why the old reasoning was wrong: the cache is +not allocated-and-untouched, it is fully in core -- 1023.8 MB allocated against +1024.1 MB resident, and 190.8 against 191.1 when sized down. + +The explanation for the original 188 MB is two sections below, in this report's +own words: `--kv-cache-memory-bytes` *was silently discarded on CPU*, because +`CpuPlatform.check_and_update_config` overwrote it from +`VLLM_CPU_KVCACHE_SPACE` unconditionally. So the "32x shrink" never happened; +both arms ran the same cache and the 188 MB was measuring something else. The +lesson is the one this study keeps relearning: a lever tested through a path +that silently ignores it measures nothing, and "no effect" is the most +dangerous result to accept without checking that the knob was connected. + +The dual weight layout the W4A16 path keeps (tinygemm for decode, a row-major +copy for prefill) *is* one. Median of three interleaved passes, anonymous RSS, +spread under 1.5%: + +| build | resident | prefill | decode | init | +|---|---|---|---|---| +| W4A8 g128 | 3640 MB | -- | 86.6 tok/s | 15.0 s | +| W4A16 g128 | 3542 MB | 3252 tok/s | 76.9 tok/s | 12.4 s | +| **W4A16, `prefill_dequant: false`** | **2802 MB** | 448 tok/s | 78.5 tok/s | 16.0 s | + +The second layout costs **740 MB** and buys **7.3x prefill**, because the +tinygemm kernel dequantizes once per row of activations and its cost is linear +in the batch. + +**That trade-off is smaller than it looked, but it is real** (2026-09-13). It +rested partly on treating the tinygemm layout as opaque, so a row-major +duplicate was kept beside it to have something to dequantize from -- but the +duplicate was also doing work: `to_split_halves` performs the nibble +de-interleave *once at load*, so every prefill call reads two contiguous +planes. It is a precomputed permutation, not just a copy. Measured on this host instead of +assumed -- by packing an index pattern and reading back where each nibble +landed -- the layout is `[N/64][K][32 bytes]`: K is the middle axis, so a K +range is contiguous within each n-block, and inside every 32-byte run byte `d` +carries output channel `d` in its low nibble and `d + 32` in its high one, the +same permutation for every `k` and every block. Decoding it is a deinterleave, +not a reverse-engineered ISA table. + +So the prefill path can dequantize from the kernel's own layout, and then the +duplicate is not allocated -- at the cost of redoing that de-interleave on +every call. The reconstruction is **bit-exact** against the previous +dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048. +Shapes where N does not tile the 64-channel block (N=80 is 64+16) keep the +copy rather than mis-read the layout; every Spark linear tiles cleanly. + +Measured with `VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT` toggling the two paths in +otherwise identical code, 3 arms x 2 interleaved passes, anonymous RSS summed +over the process tree, 1536-token context: + +| build | resident (median) | peak | spread | +|---|---|---|---| +| W4A16, second copy (old) | 4591 MB | 5508 MB | 0.2% | +| **W4A16, one layout** | **3976 MB** | **4593 MB** | 5.4% | +| W4A16, `prefill_dequant: false` | 3902 MB | 4522 MB | 2.6% | + +**615 MB of resident memory and ~915 MB of peak**, and the one-layout build is +statistically indistinguishable from `prefill_dequant: false` -- the harness +refused to rank them (0.4% apart, inside the measurement's own spread). The +arithmetic predicts 764 MB from the config; ~150 MB of that is allocator +retention `malloc_trim` does not return. + +It is not free. Prefill, pp512, pooled over 7 interleaved passes per arm on +idle node-1 cores with memory bound to the same node: + +| build | resident | pp512 median | range | spread | +|---|---|---|---|---| +| two layouts (old default) | 4591 MB | 2190 tok/s | 2117-2345 | 9.7% | +| **one layout, direct dequant** | **3976 MB** | **1344 tok/s** | 1271-1555 | 18.3% | +| `prefill_dequant: false` | 3902 MB | 453 tok/s | 451-453 | 0.3% | + +So it costs **1.63x prefill** on medians -- and the ratio itself is only good +to about 1.4-1.8x, because both dequantizing arms swing ~10-18% on this shared +host while the tinygemm-only arm reproduces to 0.3%. The *existence* and rough +size of the cost is solid (a 63% gap is far outside that band); the third digit +is not. An earlier draft of this section said 1.9x, taken from the single +surviving pass of a grid whose other passes had crashed -- a number the harness +itself had labelled "single pass -- not a result". + +The mechanism is that undoing the kernel's interleave per call costs ~1.6x +reading the pre-permuted planes (14.4 ms vs 9.2 ms per `gate_up` dequantize at +8 threads). Two variants were tried and rejected: dequantizing to `[K, N]` and +using `matmul` instead of `F.linear`, which avoids the element-wise transpose, +measured *worse* at 20.3 ms, and the blocked form needs either 208 tiny GEMMs +or the same permutation. + +What this changes is the shape of the choice, not its existence: + +* against `prefill_dequant: false` it is **strictly better** -- the same memory + to within the noise, **2.97x** the prefill, and it does not route multi-row + batches into the tinygemm/AMX SIGILL hazard documented in `cpu_int4.py`. That + setting is now dominated and should not be used. This comparison is the solid + one: the tinygemm-only arm reproduces to 0.3%. +* against the old default it is a real dial: 615 MB for ~1.6x prefill. On a + device chosen for its memory ceiling that is the right side of the trade; + on one with headroom it is not. `direct_dequant: false` in the checkpoint's + `quantization_config` selects the old behaviour -- a config field rather than + only an env var, because the two paths build different graphs and must not + share an AOT compile artifact (see below). + +Recovering the prefill would take a C++ de-interleave rather than an ATen +expression chain, which is a real kernel and not in scope here. + +One hazard found while measuring this, worth recording because it presented as +a crash rather than a wrong number. The two paths build different graphs -- one +passes `weight_halves` into the custom op, the other passes `None` -- but vLLM +keys its AOT compile cache on the *config*, so an env-var-only switch shares +one cached artifact between them. Three passes died with +`KeyError: 'weight_halves'` inside an AOT-loaded forward after the first pass +saved an artifact for the other path. The choice is therefore a +`quantization_config` field, which `ModelConfig.compute_hash` does hash; the +env override remains for experiments and needs its own `VLLM_CACHE_ROOT`. + +### The largest memory item was a RoPE table for a context nobody asks for + +The report used to account for memory by subtraction -- weights, imports, and +an "unexplained ~1.6 GB" left over. A remainder computed from two uncertain +terms is not a measurement, so `memory_ledger.py` counts instead: unique tensor +storages keyed by `data_ptr` (so tied weights are counted once), the KV cache +found by attribute, and every `/proc/self/smaps` mapping bucketed. Allocated is +separated from resident with `mincore(2)`, because a cache that was allocated +and never written costs address space, not memory, and counting it would hide +the remainder rather than explain it. + +The first run put 640 MB in an "other" bucket across 114 tensors. Naming the +largest storages showed what it was: + + model.layers.0.self_attn.rotary_emb.cos_sin_cache 512.0 MB [1048576, 256] + model.layers.3.self_attn.rotary_emb.cos_sin_cache 128.0 MB [1048576, 64] + +Spark advertises a **1 048 576-token context**, and `spark2_5.py` passed +`max_position=config.max_position_embeddings` straight into `get_rope`, so vLLM +built `torch.arange(1048576)` cos/sin tables -- two of them, because the +sliding and full layers are trained with different theta and different partial +rotary factors, giving one 256-wide table and one 64-wide one. Both fully +resident by `mincore`. For a deployment whose `max_model_len` is 2048 they need +**1.25 MB between them**. + +vLLM's scheduler cannot produce a position beyond `max_model_len`, so the table +only has to cover that. Capping it, 3 interleaved passes per arm with +`VLLM_OMNI_SPARK_ROPE_FULL` toggling the two: + +| | settled | peak | init | +|---|---|---|---| +| 1M-position tables | 3891 MB | 4019 MB | 13.5 s | +| **capped to `max_model_len`** | **3273 MB** | **3370 MB** | 13.7 s | + +**618 MB settled and 649 MB of peak, for one line and no measurable init +cost** -- the largest single item found in this study, larger than the single +weight layout (615 MB) or the blocked repack (489 MB), and unlike those two it +costs nothing at all. The ledger's storage accounting agrees independently: the +"other" bucket falls from 640.0 MB to 1.3 MB and unique tensor storage drops +638.8 MB. + +Two things the same ledger exposes, not yet acted on: + +* **The KV cache is fully resident** -- 1023.8 MB allocated, 1024.1 MB resident + by `mincore`. The "not a memory lever" verdict below was measured while + generating 8 tokens; at page level the cache is entirely in core, and it is + 27% of total memory. That conclusion needs re-opening on this evidence. +* **`embed_tokens.weight_packed` (128 MB) is kept only because the tinygemm + layout was believed un-addressable for a single-row lookup.** That premise is + now known to be false -- the deinterleave above reaches any row -- so the + row-major copy is probably removable too. + +### Peak load memory: the number an edge device actually has to survive + +Everything above reports memory *after* the allocator is asked for its arenas +back. That is the wrong target for a device: `release_after_load()` runs only +once `load_model` and warm-up have both returned, so a box that cannot hold the +peak never reaches the release. Sampling `RssAnon` on a timer through the load +(rather than reading the endpoint) put the W4A16 peak at **5722-5837 MB** +against ~4480 settled -- a 1.2-1.3 GB spike nothing in the tree had measured. + +Most of it was in our own repack path. `unpack_nibbles` expands 4-bit codes to +int32 -- 4 bytes per weight, eight times the packed form -- because the +kernel's packer wants int32, and doing that for a whole matrix makes the +transient the largest object in the process: + +| tensor | shape | int32 expansion | +|---|---|---| +| `gate_up` | 13312 x 2048 | 104 MiB | +| `down_proj` | 2048 x 6656 | 52 MiB | +| `q_k_v_proj` | 3072 x 2048 | 24 MiB | +| **embedding (tied, so also the head)** | **131072 x 2048** | **1024 MiB** | + +against 24 MiB of packed weights for an entire layer. + +The packed layout is `[N/64][K][32 bytes]`, so n-blocks are independent and the +expansion can cover a few thousand rows at a time -- verified byte-identical to +packing the whole matrix at four shapes plus a ragged last chunk. Measured with +`VLLM_OMNI_CPU_INT4_REPACK_ROWS` toggling the two in identical code, 3 +interleaved passes, node-local cores: + +| | peak | settled | load-time transient | init | +|---|---|---|---|---| +| whole-matrix repack | 4502 MB | 3905 MB | 550 MB | 17.7 s | +| **blocked repack** | **4013 MB** | 3892 MB | **105 MB** | 16.6 s | + +**489 MB of peak, for nothing.** Settled memory and init time are unchanged, +and the repack is in fact *faster* chunked than whole -- 79 ms against 208 ms +for the embedding at 24 threads, because the working set stays in cache. The +chunk size is tuned from that sweep: 2048 rows is both the fastest and nearly +the leanest (16 MiB transient), where 512 costs 116 ms and 32768 costs 129 ms. + +Two things this nearly got wrong, both caught by re-measuring rather than +reasoning. The first blocked pass showed init at 50.2 s against 17.7 -- but it +is the one pass the harness flagged `[busy]` at load 34.8, and the two clean +passes read 16.6 and 16.2. And its settled memory looked 239 MB worse, which +was pass-1 scatter: the medians are 3892 against 3905. + +The absolute peak improves less than the 1 GB the arithmetic predicts, because +the single-layout change above had already removed `to_split_halves`, which was +doing its own whole-matrix expansion. The two items overlap and their savings +must not be added. + +### Kernel fusion: a negative result + +vLLM's CPU build ships `_C::rms_norm` and `_C::fused_add_rms_norm`, and +`vllm/kernels/vllm_c.py` already wraps both as IR-op implementations -- but +gates them on `GPGPU_DEVICE = is_cuda_alike() or is_xpu()`, so on CPU every +norm falls through to the PyTorch path. In isolation the fused kernels are +**3.6x and 4.5x faster** at Spark's decode shape (5.76 vs 20.99 us; 9.24 vs +41.14 us). End to end they are **slightly slower**: 72.7 vs 74.0 tok/s. + +The reason is that the native norm was never running alone -- inductor fuses it +into the residual add and the surrounding elementwise work, and an opaque `_C` +call breaks that fusion for more than the norm itself saves. +`vllm_omni/model_executor/layers/cpu_ir_ops.py` keeps the registration, off by +default (`VLLM_OMNI_CPU_FUSED_NORMS=1` to try it on another machine), with the +measurement recorded. `gelu_and_mul` is deliberately not registered at all: its +fused CPU kernel is 2.7x *slower* than native at Spark's GEGLU shape (64.11 vs +23.36 us), and `GeluAndMul.forward_cpu` in vLLM only reaches for it on POWERPC +anyway. + +Two more fusion attempts that lost, recorded so they are not repeated: +`cpp.min_chunk_size` raised to keep inductor's small elementwise kernels +single-threaded cost 6% of decode and 19% of prefill; and the head gate written +as an inline reduction (rather than behind an opaque op) made the whole step +**1.8x slower** -- 78.8 to 44.5 tok/s -- because inductor's generated loop is +worse than the matmul it replaced. + +## 3. GPU: vLLM-Omni vs SGLang + +One L20X (140 GiB HBM), bf16, single sequence, caches disabled on both +(`enable_prefix_caching=False` / `disable_radix_cache=True`), 2048-token +prefill so the measurement is not dominated by dispatch overhead. + +| engine | init | pp2048 | tg128 | +|---|---|---|---| +| vLLM-Omni | 111.3 s | 137 102 tok/s | **451.4 tok/s** | +| SGLang 0.5.19 | 97.1 s | **142 253 tok/s** | 434.7 tok/s | + +A tie within 4% either way: SGLang +3.8% on prefill, vLLM +3.8% on decode. + +SGLang has an Intel AMX CPU backend, but the released `sgl-kernel` wheel does +not ship the CPU ops it needs (`convert_weight_packed` is absent), so SGLang +could not be measured on CPU without a source build. On the two targets this +goal is actually about -- CPU and a phone -- SGLang does not run at all today. + +## 4. Galaxy S25 (Snapdragon 8 Elite), real device + +`vllm_omni/edge/spark_export.py` exports one decode step with the KV cache as +explicit tensors, sized per layer type: sliding layers get a fixed window at +any context, full layers grow. Parity against the HF reference is **140.3 dB** +with identical top-1 and top-5. + +### What the device profile said to fix + +AI Hub returns per-op cycle counts, and they made the first export's mistake +obvious. Of a sliding layer at w4a16: + +| op | share | what it was | +|---|---|---| +| `Output` | 9.9% | handing the recomputed 512-entry window back | +| `output_..._converted` | 3.1% | quantising that window on the way out | +| `Slice` x3 | 7.4% | rolling the window to drop the oldest entry | +| **subtotal** | **20.4%** | **moving a cache-sized tensor across the graph boundary, once per token, in 21 of 28 layers** | + +Full-attention layers spent a similar 9.5% on `Concat`, appending the new key +to the cache. + +The fix is a ring buffer: the runtime owns a fixed-size cache and writes the +new entry into the slot for this position, so the graph returns one K/V entry +instead of a window and never rolls anything. Ring order needs no fixing up -- +softmax over keys is order-independent and each cached key already carries its +own rotary phase, so only slots not yet written need masking. The sliding +cache correspondingly holds `window - 1` entries, because a 512-token window +counts the current token (transformers' own sliding cache holds 511 for +exactly this reason). + +| layer | roll | ring | | +|---|---|---|---| +| sliding, fp16 | 1.689 ms | **1.486 ms** | −12.0% | +| sliding, w4a16 | 0.917 ms | **0.838 ms** | −8.6% | +| full, fp16 | 1.559 ms | 1.567 ms | no gain | + +Full layers gain nothing, so the default now picks per layer type: ring for +sliding, roll for full. + +### Three things that did not work, and one that changed the answer + +**The concat was not the problem.** Joining in score space instead of key +space removes the concat entirely and is numerically identical (140.3 dB), but +it needs an explicit max/exp/sum/div in place of a fused `Softmax`, and that +cannot be quantised -- QNN's context-binary converter exits 14 at w4a16 while +every fp16 build compiles. Two other shapes failed the same way: a broadcast +multiply plus `ReduceSum` for the new token's score, and a two-axis broadcast +multiply for its value contribution. Keeping roll's fused-softmax attention and +changing only what the layer *returns* captures the win and compiles. + +**The cheaper GELU is not cheaper here.** The erf form is 11–13% of every +layer. The tanh approximation holds 65.2 dB with top-1/top-5 intact, so it was +adoptable -- but on this NPU it is *slower* (1.535 vs 1.483 ms). The exact form +stays, which is also the better-fidelity choice. + +**Per-graph overhead is negligible, so the per-layer method is sound.** +Profiling one layer at a time pays the graph's Input/Output boundary once per +layer, where a real step pays it once for 28. Exporting the architecture's +repeating 4-layer block (3 sliding + 1 full) measured 6.103 ms against 6.016 ms +summed from single layers — a 1.4% gap. There is no hidden win in fusing +layers, and extrapolating from per-layer numbers is legitimate. + +**Uncalibrated quantisation produces garbage, which invalidates the earlier +numbers.** Running the real layer on the device and comparing against the fp32 +reference: + +| build | SNR vs reference | +|---|---| +| fp16 | 60.6 dB | +| w4a16, no calibration data | **−1.6 dB** | +| w4a8, no calibration data | **−1.28 dB** | + +Negative SNR means the output is uncorrelated with the truth. Every w4a16 +figure in the first version of this document came from a build in that state. +Quantisation ranges have to be inferred from activations the model actually +sees: real ones have `x` at absmax 0.12 and the KV cache at 7.4, where the +random tensors used for tracing had 0.4 and 2.5. + +Re-quantising with activations captured from the reference model on real text +(4 samples, English, code and Chinese) fixes it: + +| scheme | SNR | sliding layer | +|---|---|---| +| **w8a16, calibrated** | **36.9 dB** | **0.803 ms** | +| w16a16, calibrated | 44.7 dB | 1.656 ms | +| w8a8, calibrated | 5.1 dB — broken | 0.749 ms | +| w4a16 / w4a8, calibrated | converter fails | — | + +int8 *activations* destroy the layer even with calibration, which is the third +model family in this repo to show it after Qwen3-TTS and MiniCPM-o. int4 +weights do not survive the calibrated path at all. **w8a16 with real +calibration data is the deployable configuration**, and it is both faster and +verified where the old w4a16 was neither. + +### What is left, and why int4 is not it + +Profiling the optimised layer shows the shape of the problem has changed +completely. `Input` and `Output` -- 26% of the original layer -- are no longer +in the top ten ops at all. What remains, of 0.803 ms: + +| op group | share | can it move? | +|---|---|---| +| MLP up + down matmuls | 36.3% | no, this is the model | +| GELU (fused elementwise) | 17.5% | no, tanh is slower here (measured) | +| attention matmuls | 10.5% | no | +| RMSNorm, decomposed | 8.5% | **yes, see below** | +| cache slice | 4.1% | marginal | + +**int4 weights are not the next lever, on two independent counts.** They do not +compile: AI Hub's calibrated quantise path fails with `QAIRT converter failed +with exit code 1` for w4a16 and w4a8, with and without `--quantize_io`. And +they would not help anyway -- on this NPU 4-bit and 8-bit weights time +identically, for the layer (0.917 vs 0.912 ms) and for the output head (3.868 +vs 3.862 ms). The calibrated *8-bit* build is faster than the uncalibrated +4-bit one (0.803 vs 0.838). 4-bit buys memory here, not speed. + +**The real remaining item is RMSNorm fusion, which calibration costs us.** The +uncalibrated build contains two fused `rms` ops worth 1.0% of the layer; the +calibrated build contains eight decomposed `Pow`/`ReduceMean`/`Div`/`Mul` ops +worth 8.5%. Inserting QDQ nodes around each op breaks QNN's RMSNorm pattern +match. The calibrated build is still faster overall and is the only correct +one, so this is a trade worth making today -- but roughly 7.5% of every layer +is sitting there for whoever can keep the fusion under quantisation. +(`F.rms_norm` is not a route to it: the TorchScript ONNX exporter has no +`aten::rms_norm` at any opset.) + +### The decode step, end to end + +Every component measured on a real S25 at calibrated w8a16, 1k context: + +| component | ms each | ms total | SNR | +|---|---|---|---| +| sliding layer x21 (ring) | 0.803 | 16.86 | 36.9 dB | +| full-attention layer x7 (roll) | 0.942 | 6.59 | 37.4 dB | +| output head x1 | 3.907 | 3.91 | 62.8 dB | +| **decode step** | | **27.36** | **36.5 tok/s** | + +Against the previous best (roll layout, uncalibrated w4a16, plus the same +head) at 31.01 ms: **11.8% faster, and numerically valid for the first time.** + +The output head is 14% of the step and is 98.6% a single 2048x131072 matmul +tied to the embedding — there is no overhead in it to remove. It was missing +from the first version of this document entirely. + +At 4k context the full-attention layers grow (2.96 ms each at fp16 vs 1.56 at +1k) while the sliding layers do not move at all, which is what the hybrid +architecture buys: + +| quant | context | Spark hybrid | all-full counterfactual | gain | +|---|---|---|---|---| +| fp16 | 1 k | 46.4 ms | 43.7 ms | 0.94x | +| fp16 | 4 k | 56.2 ms | 83.0 ms | 1.48x | + +At 1k in fp16 the hybrid is *slightly slower*: sliding layers rotate all 256 +head dims where full layers rotate 64, and below roughly 2k that RoPE cost +outweighs the shorter cache. It pays from ~2k onward and keeps paying, because +the full-layer term grows with context and the sliding term never does. + +### Which phone runtime, and which processor + +The same exported sliding layer, one Galaxy S25: + +| runtime | processor | precision | ms/layer | +|---|---|---|---| +| **QNN context binary** | **Hexagon NPU** | **w8a16 calibrated** | **0.803** | +| TFLite | CPU | int8 | 0.877 | +| QNN context binary | Hexagon NPU | fp16 | 1.479 | +| TFLite | Hexagon NPU | fp16 | 1.491 | +| ONNX Runtime | Hexagon NPU | fp16 | 1.500 | +| TFLite | Adreno GPU | fp16 | 2.259 | +| TFLite | CPU | fp16 | 2.958 | + +Two readings. At matched fp16 the three NPU runtimes land within 1.4% of each +other, so the export layout -- static shapes, explicit caches, GQA without +expanding K/V -- is what sets on-device speed, not the engine; that reproduces +what the Qwen3-TTS talker step showed on the same device. And the placement is +the opposite of that TTS result: Spark's decoder wants the NPU (2.0x the CPU, +1.5x the GPU) because it is matmul-bound, where the TTS vocoder wanted the GPU +because it was elementwise-bound. + +The row that matters for competition is the int8 CPU one. llama.cpp cannot be +run on an AI Hub device, and on a phone it would use 4-bit weights on the CPU, +so comparing our NPU path against an fp16 CPU run would flatter us. The +closest honest proxy AI Hub offers is the same graph, int8, on the S25's own +CPU: 0.877 ms against our 0.803 ms, so the NPU path is **9.2% ahead of the +processor and precision class llama.cpp would actually run in**. That is a +proxy, not llama.cpp itself. + +Two independent checks say the proxy is not flattering us either. A 1.11 GiB +Q4_K_M build on the S25's ~137 GB/s LPDDR5X has a hard bandwidth ceiling of +7.9 ms/token at 100% of peak and 13.2 ms at a more realistic 60% — against our +measured 27.4 ms for a step that includes the output head the bandwidth model +ignores. Per layer, the bandwidth bound gives roughly 0.28-0.47 ms where the +measured int8 CPU run takes 0.877 ms, i.e. real CPU execution lands at about a +third of the bandwidth ceiling, which is the usual outcome on a phone. + +## 5. Driving an Android phone + +Spark-X2.5 is text-only, so any harness must drive from the accessibility tree +rather than screenshots. Surveyed: + +| harness | transport | text-only a11y mode | local OpenAI endpoint | role | +|---|---|---|---|---| +| droidrun/mobilerun | ADB + Portal accessibility service | yes | yes, OpenAI-compatible | control | +| ghost-in-the-droid/android-agent | ADB + uiautomator, no app on device | yes | lists vLLM, base_url undocumented | control | +| mobileClaw | on-device app, accessibility | vision-oriented | n/a | control | +| AndroidWorld (DeepMind) | Android emulator | yes | via custom agent | **evaluation**, 116 scored tasks | + +Recommendation: droidrun/mobilerun for control (it is the only one whose +OpenAI-compatible base URL is documented, so it points at a local vLLM server +unchanged), AndroidWorld for measuring whether the agent actually succeeds. + +`analysis/experiments/spark_edge/android_agent/` implements the action space +those harnesses share -- `tap`, `type_text`, `swipe`, `press_key`, `open_app`, +`task_complete`, all indexed against accessibility-tree elements, with the +`adb shell input` command each maps to -- and runs Spark against ten Android +screens through vLLM's OpenAI server. + +Serving works with vLLM's stock parsers: Spark emits +`namekv`, +which is GLM-4.5's format, so `--tool-call-parser glm45 --reasoning-parser +glm45 --enable-auto-tool-choice` parses it as-is. No new parser needed. + +| config | correct tool | fully correct action | median decision | +|---|---|---|---| +| thinking on | 7-8 / 10 | 5-6 / 10 | 8.9 s | +| thinking off | 5 / 10 | 4 / 10 | **0.61 s** | + +(16 CPU cores, bf16, measured under concurrent benchmark load, so latency is +pessimistic.) + +Thinking is worth roughly two of ten decisions and costs 14.6x the latency. +With thinking off the model collapses toward answering `tap` for everything. +With it on, the failure mode is the opposite: on the one genuinely ambiguous +screen (reach Developer options, which is not visible and needs scrolling) it +reasoned past a 1200-token budget without ever closing ``, so vLLM's +reasoning parser -- which only emits reasoning once the block closes -- +returned empty content and no tool call at all. A phone agent on this model +needs a step budget and a fallback, not just a bigger cap. + +The clearest wrong action was tapping an already-on Wi-Fi switch instead of +calling `task_complete`, i.e. undoing the task it had completed. + +No Android device or emulator is attached to this host (`/dev/kvm` absent), so +these numbers measure the model's decisions over real accessibility-tree +layouts, not a live device. + +### 5b. Driving it in a loop, not one decision at a time + +The numbers above measure single decisions over fixed screens. A phone agent +is a loop, and a loop fails differently. `analysis/experiments/spark_edge/ +android_agent/` is now a working toolkit -- uiautomator XML to an indexed +element list, an ADB device and a scripted simulator that emits the same XML, +a tool executor, a tolerant tool-call parser and the loop itself (21 tests, +no phone and no server required). + +**Stock `glm45` does not always parse Spark.** Earlier this note said Spark's +tool format is GLM-4.5's, so vLLM's parsers work as-is. True on 4 of 5 screens. +On the Wi-Fi toggle screen -- the one where a switch's state decides the +action -- the model left the format and emitted + + [tap] + {"tool": "tap", "parameters": {"index": 1} + +unterminated, with `tool_calls` empty. An agent reading only `tool_calls` sees +a step with no action. `tool_parse.py` recovers the bracketed form, GLM blocks +and bare JSON, repairing unbalanced braces, and reports every recovery rather +than silently repairing. + +**Navigation survives quantization; recognising completion does not.** Same +harness, same simulated phone, one run per cell: + +| build | decode | "Turn on Wi-Fi." | "Set an alarm for 7:30 AM." | +|---|---|---|---| +| bf16 | 41.0 tok/s | done in 4 steps | wrong app (Settings, not Clock), toggles Airplane mode, stuck | +| **W4A16 g128** | 80.0 tok/s | done in 4 steps | right app, full navigation, then no tool call | +| W4A8 g128 | 86.6 tok/s | turns Wi-Fi on, then taps it off again (2 runs) | reaches Alarm, then loops | + +Every build walks home -> Settings -> Network -> Wi-Fi correctly. Only some +notice the switch now reads `checked=true` and stop. W4A8 -- the fastest build, +and the one a speed-first selector picks -- ends with the task undone. + +This is what section 2c's fidelity numbers meant. W4A8 reproducing 1/12 greedy +prompts against its own bf16, and W4A16 2/12, looked like a small difference on +a proxy metric; it is the difference between finishing a task and reversing it. +**For agentic work the 4-bit choice is W4A16**, which `weight_path.py` selects +under `priority="fidelity"` -- 8% of decode for the difference. + +Adding an explicit "if the screen already shows what was asked, call +task_complete" rule to the system prompt did not rescue W4A8. Nor did feeding +back "the screen did not change", which bf16 was told five times on the alarm +task before swiping again. + +Limits: two tasks, one run each, on a simulated phone -- enough to show a +failure mode reproduces, not to rank builds by success rate. The simulator +emits real uiautomator XML, so the parser, the index-to-coordinate mapping and +the stopping rules run on the path hardware drives; whether a real Settings app +is laid out the way these fixtures assume is still untested, because this host +has no `adb`, no emulator and no `/dev/kvm`. + +## 6. Where it stands against the goal + +- **Runs on CPU**: yes. Faster than llama.cpp at matched precision on both + prefill (8.7-15.4x) and decode (1.19-2.65x), and an int8 path adds a further + 1.6x. +- **Runs on Galaxy S25**: yes, measured on real hardware -- a full decode + step, output head included, is 27.4 ms at 1 k context (36.5 tok/s) in + calibrated w8a16, with every component verified at 36.9-62.8 dB against the + reference. That is 11.8% faster than the first working version and the first + configuration whose output is actually correct: the uncalibrated w4a16 builds + quoted earlier score -1.6 dB, i.e. uncorrelated with the truth. +- **Beats every edge engine**: on prefill, yes, against everything that can + run the model, at any precision -- 4.7x over llama.cpp's best quantized + build at 32 threads. On decode it depends on load: llama.cpp Q4_K_M wins + single-stream by 1.6x, vLLM int8 wins from batch 16 and leads 1.72x at batch + 64. The single-stream gap is not closable on vLLM 0.28's CPU backend, which + ships no dense int4 kernel. +- **Beats SGLang**: on decode by 3.8% on GPU (prefill 3.8% the other way), and + by default on CPU and phone, where SGLang does not run -- its released + `sgl-kernel` wheel has no CPU ops and it has no phone backend at all. +- **Fastest on the phone**: of every runtime and processor combination + measurable on the S25, our QNN NPU path is quickest, including a 9.2% margin + over the int8-on-CPU class llama.cpp would run in. llama.cpp itself cannot be + run on an AI Hub device, so that specific comparison is a proxy. +- **Phone control**: the action space, an agent loop and measured decision + accuracy are in `analysis/experiments/spark_edge/android_agent/`; serving + needs no new parser, just `--tool-call-parser glm45 --reasoning-parser + glm45`. + +Open items: no dense int4 on CPU (upstream kernel gap); int8 long-context +fidelity would benefit from a smarter scheme than round-to-nearest; on the NPU +the next ~7.5% per layer is RMSNorm fusion, which calibration currently breaks +(int4 weights are neither available nor useful there -- 4-bit and 8-bit time +identically on this hardware); and none of the phone-agent numbers have been run against a live device, since +this host has no emulator (`/dev/kvm` absent). diff --git a/docs/analysis/tts_edge_design.md b/docs/analysis/tts_edge_design.md new file mode 100644 index 000000000..152a60327 --- /dev/null +++ b/docs/analysis/tts_edge_design.md @@ -0,0 +1,148 @@ +# Edge TTS design (vLLM-Omni-compatible) + +Scope: run vLLM-Omni's autoregressive-plus-codec TTS pipelines on edge devices with the same chunking, state, and interruption semantics as the server, using llama.cpp, MNN, or ExecuTorch as the execution runtime. Reference model: **Qwen3-TTS-12Hz (0.6B / 1.7B)**, because it is the only family implemented natively in vLLM-Omni, llama.cpp, and MNN at the analyzed commits [Verified: [engines/vllm_omni_models.md](engines/vllm_omni_models.md) §2, [engines/llama_cpp.md](engines/llama_cpp.md) §9, [engines/mnn.md](engines/mnn.md) §9]. Secondary references: MOSS-TTS (transformer-only codec with ring KV), CosyVoice3 (flow-matching code2wav), Voxtral TTS (ExecuTorch streaming runner). + +## 1. Analysis of the pipeline being ported + +### 1.1 Conditioning + +| Input | Qwen3-TTS (server) | Edge implication | +|---|---|---| +| Text | Tokenized by the HF tokenizer; the task template (CustomVoice / VoiceDesign / Base) is built inside the model by `Qwen3TTSPromptEmbedsBuilder.build_prompt_embeds` in `prompt_embeds_builder.py`, called from the talker; the example only supplies `additional_information` and estimates prompt length [Verified] | Tokenizer must ship on device (llama.cpp vocab, MNN `.mtok`, ExecuTorch tokenizers submodule). The prompt-embeds builder is engine code and the parity source for the edge orchestrator's template construction. | +| Speaker | ECAPA speaker encoder on reference audio (`qwen3_tts_talker.py#L202`) or predefined speaker id; optional in-context codes from a Mimi encoder [Verified] | llama.cpp and MNN both run the ECAPA encoder on device (`qwen3tts-spkenc.cpp`; MNN requires `--ref_audio`, zero-embedding path unsupported [Verified]). Speaker embeddings should be **computed once and cached** as a small vector; the encoder does not need to be in the hot path. | +| Instruction / style | Text tokens in the template [Verified] | Same as text. | + +### 1.2 Generation type + +Autoregressive over codec frames at **12.5 Hz** (80 ms per frame): the talker predicts codebook 0 per step and a 5-layer code predictor fills the remaining 15 codebooks, re-prefilling per codebook without a KV cache [Verified, `common/qwen3_code_predictor.py#L996`]. In vLLM-Omni the code predictor is captured in dedicated CUDA graphs (`talker_mtp`); in llama.cpp it runs with on-graph top-k/top-p [Verified]. There is no diffusion in this family; CosyVoice3 (CFM, 10 Euler steps, CFG batch 2) and IndexTTS-2 (S2Mel DiT) are the diffusion-flow variants and cost proportionally more per chunk [Verified]. + +### 1.3 Codec / vocoder + +`Qwen3TTSTokenizerV2Decoder`: RVQ embedding sum → causal conv → 8-layer sliding-window (72 frames) transformer → 2×2 upsampling → BigVGAN/Snake stack, fp32, 24 kHz, 1920 samples per frame [Verified]. Cross-chunk state: conv context (2 + 12 frames), a reference-prefix `DynamicCache`, and the 72-frame attention window (`modeling_qwen3_tts_tokenizer_v2.py#L803`) [Verified]. On the server this runs as an `LLM_GENERATION` stage with stateful CUDA graphs per reachable chunk length. + +### 1.4 Streaming and first-audio latency structure + +Server schedule (from `deploy/qwen3_tts.yaml` and measured in [experiments.md](experiments.md)): + +``` +talker step 1 ──► 1 frame ──► code2wav ──► first chunk (1,920 samples) TTFA ≈ 28 ms on L20X +talker steps 2..26 ──► 25 frames ──► code2wav with 72-frame left ctx ──► chunk 2 (48,000 samples) +160 ms +... +EOS ──► remaining frames ──► final message carries the tail +``` + +TTFA is bounded below by: text prefill + one talker decode + code predictor (15 sub-steps) + decoder on 1 frame with empty context + transfer. On the server the transfer is a shared-memory hop; on device it is a function call. + +### 1.5 Real-time factor, quality, interruption, memory, thermals + +| Concern | Server fact | Edge design consequence | +|---|---|---| +| RTF | 0.08–0.10 on L20X (measured); repo-reported 0.16 on H200 at concurrency 1 | Per-frame budget on device is 80 ms for talker + predictor + decoder amortized. From the CPU baseline (409 M LLM at 135 tok/s Q4_0 on 4 x86 threads, measured), the AR part of a 0.6B talker is plausible on ARM big cores; the decoder (BigVGAN + 8-layer transformer per 25-frame window) is the unknown and must be measured first [Unknown]. | +| Quality | fp32 decoder, BF16 talker; quantization "scoped to the AR LM stage" in repo docs | Quantize the talker (Q4/Q8, W4 block-64); keep the decoder fp16 at most; validate with the parity harness (§5). | +| Interruption | `AsyncOmni.abort` → Orchestrator → per-stage abort; decoder state released in `on_requests_finished` [Verified] | Edge orchestrator needs the same three-part abort: stop talker loop at the next frame, drop queued codec frames, flush/reset decoder state; playback ring buffer must be cleared by the app. | +| Memory | Stage 0 asked for 30 % of a 140 GB card by config, not need [Unknown actual] | Weights: 0.6B talker ≈ 0.4 GB at Q4, ≈ 1.2 GB at fp16 [Inferred from parameter count]; decoder ≈ 200 M params (repo doc for Qwen3-Omni code2wav) ≈ 0.4 GB fp16 [Inferred]; KV for ≤4096 positions small. Target under 2 GB resident for the 0.6B path. | +| Thermals | N/A on server | Sustained decode on phone big cores throttles within minutes; design must allow moving the decoder to GPU/NPU (MNN OpenCL/QNN, ExecuTorch CoreML/QNN) and keep the talker on CPU, or vice versa, without changing the orchestrator. | + +## 2. Design + +### 2.1 Architecture + +```mermaid +graph TB + subgraph app["Application (Android/iOS/Linux)"] + UI["Text + speaker request"] --> ORCH + ORCH["Edge TTS orchestrator (C++)
mirrors PipelineConfig: talker → code2wav
chunk schedule 1 / 25 / ctx 72
abort + backpressure"] + ORCH --> RING["Audio ring buffer
(owner: app)"] + RING --> OUT["Audio sink"] + end + subgraph adapters["Adapter interfaces"] + TA["ITalkerAdapter
prefill(tokens, spk) / step() → 16 codes"] + DA["IDecoderAdapter
decode(frames, is_first, is_last) → PCM
reset()"] + SA["ISpeakerAdapter
embed(wav) → vector (cached)"] + end + ORCH --> TA + ORCH --> DA + ORCH --> SA + subgraph rt["Runtime bindings (choose per target)"] + L1["llama.cpp: libmtmd Qwen3-TTS graphs
(CPU/Metal/OpenCL/Hexagon/CUDA)"] + M1["MNN: Llm talker + code_predictor + speech_decoder modules
(CPU/OpenCL/QNN/Metal/CoreML)"] + E1["ExecuTorch: talker.pte + predictor.pte + decoder.pte
(XNNPACK/CoreML/QNN/CUDA)"] + end + TA --> L1 + TA --> M1 + TA --> E1 + DA --> L1 + DA --> M1 + DA --> E1 +``` + +### 2.2 Adapter interfaces [Proposal] + +```cpp +// Talker: one AR frame per step; returns 16 codebook ids (codebook 0 from the LLM, 1..15 from the predictor) +struct ITalkerAdapter { + virtual bool prefill(const std::vector& text_tokens, const SpeakerCond& spk) = 0; + virtual bool step(std::array& codes, bool& eos) = 0; // ≤ 80 ms budget + virtual void set_sampling(const Sampling& s) = 0; // temp 0.9, top_k 50 (deploy default) + virtual void reset() = 0; // drops KV, ready for next utterance +}; +// Decoder: stateful across chunks; caller decides chunking (1, then 25) and supplies left context policy +struct IDecoderAdapter { + virtual bool decode(const int32_t* codes, int n_frames, bool first, bool last, PcmBuffer& out) = 0; + virtual void reset() = 0; // clears conv ctx, window, prefix cache + virtual int left_context_frames() const = 0; // 72 for Qwen3-TTS +}; +struct ISpeakerAdapter { virtual bool embed(const float* pcm, int n, int sr, std::vector& out) = 0; }; +``` + +Orchestrator loop (per utterance). The chunk schedule is a parameter mirrored from the deploy YAML, not a constant: the measured default (1 frame, then 25) gives an ≈80 ms first chunk followed by a ≈160 ms wait, i.e. a playback underrun if playback starts on the first chunk; the YAML's `codec_chunk_ramp` (for example 4, 4, 8, 16, 25) or `codec_chunk_adaptive` remove that cliff and should be the edge default. Gate on uninterrupted-playback latency (time until playback can start without a later gap), measured separately from first-chunk arrival. + +1. `spk = cache.get_or(ISpeakerAdapter::embed)`; `talker.prefill(tokens, spk)`. +2. Run `talker.step` until `schedule.size(0)` frames are collected (1 frame in the measured baseline; 4 with the example ramp) → `decoder.decode(frames, first=true)` → push PCM (this is the TTFA path). +3. For chunk index i ≥ 1, accumulate `schedule.size(i)` frames (ramp or adaptive, converging to 25; or until EOS) → `decoder.decode(frames, first=false, last=eos)` → push. `schedule` mirrors `codec_chunk_ramp`/`codec_chunk_adaptive`/`codec_chunk_frames` from the deploy YAML (`chunk_size_utils.py`, `ramp_chunk_size`), and TTFA is compared against a server reference run with the same schedule; 1→25 stays the measured baseline. +4. On abort: set flag checked before every `talker.step`; discard accumulated frames; `decoder.reset()`; `talker.reset()`; app clears the ring buffer. +5. Backpressure: if the ring buffer holds more than N seconds (default 2 s, one chunk), pause the talker loop rather than running ahead; this bounds memory and makes interruption cheap. + +### 2.3 State and buffer ownership + +| State | Owner | Lifetime | Notes | +|---|---|---|---| +| Talker KV cache | Runtime (llama.cpp `llama_kv_cache`, MNN `CPUKVCacheManager`, ExecuTorch in-graph buffer) | Utterance | Sized for prompt + max frames (deploy `max_model_len 4096`); reset per utterance. No paging needed at batch 1. | +| Code predictor scratch | Runtime | Step | No KV (re-prefill per codebook); llama.cpp/MNN implement this already. | +| Decoder conv/window/prefix state | Decoder adapter | Utterance | Must survive between `decode` calls; `reset()` on abort or EOS. | +| Frame accumulator (≤25 × 16 ints) | Orchestrator | Chunk | Tiny; discarded on abort. | +| PCM ring buffer | Application | Session | Sized to 2–4 s; cleared on abort; the only buffer the audio thread touches. | +| Speaker embedding cache | Application | Session | Keyed by reference audio hash. | + +### 2.4 Stage placement per target [Proposal] + +| Target | Talker | Decoder | Runtime | Rationale | +|---|---|---|---|---| +| ARM Linux board / phone CPU | CPU Q4 (KleidiAI) | CPU fp16 | llama.cpp or MNN | Both have the full path today; MNN adds NNAPI/QNN escape hatch. | +| Apple Silicon | Metal (llama.cpp) or CPU | Metal (llama.cpp) / CoreML (ExecuTorch) | llama.cpp first; ExecuTorch for ANE | llama.cpp ships xcframework; ExecuTorch has CoreML/MLX delegates. | +| Android with Qualcomm NPU | CPU or Hexagon HTP (llama.cpp) | QNN (MNN or ExecuTorch) | MNN (single engine) | MNN has QNN online/offline compile and an Android app to host it. | +| Jetson | CUDA (llama.cpp) or vLLM-Omni as-is | CUDA | vLLM-Omni (S1) or llama.cpp | S1 gives the reference numbers on the same box. | + +## 3. Hybrid (edge/cloud) variant + +Talker in vLLM-Omni (cloud or Jetson server), decoder on device. The wire payload is the codec token stream: 16 × int16 per frame at 12.5 Hz = 400 B/s of raw payload before framing and transport overhead, far below any audio stream; the on-device decoder needs only the 72-frame context it already keeps [Inferred]. Integration is more than an adapter: `serving_speech_stream.py`'s `_generate_and_send` consumes PCM from the full pipeline, so a codec-token mode needs a defined wire contract (stage termination after the talker, frame ordering, decoder state/revision, EOS and cancel), most naturally as a talker-only deploy profile plus a new stream message type; the `OmniPayload` key `codes` exists for the intra-pipeline representation [Verified]. S4-first is conditional on prototyping this contract and measuring the on-device decoder. This variant gives server-quality talker sampling, device-local audio with no PCM over the network, and a clean interruption story (client stops consuming; server abort as today). + +## 4. Runtime-specific integration notes + +- **llama.cpp**: the audio-gen helper (`mtmd-helper-gen.cpp`, `step_prompt/step_gen/get_output`) is already frame-stepped with caller-owned state; `tools/tts/tts.cpp` shows the loop. Work: expose chunked decode (1/25 frames) instead of one WAV, add abort check per step, add a server endpoint later. The decoder window (72 frames) is hard-coded in `clip.cpp` per the report; keep it in sync with the deploy YAML. +- **MNN**: `Llm::generateTTS` runs the decoder once per utterance; work: split `speech_decoder` invocation into chunked calls carrying conv/window state (the Qwen2.5-Omni path already streams with a callback and worker threads, so the pattern exists in `omni.cpp`). Export via `llmexport.py` (W4 default) and verify the fp32 requirement of the decoder on GPU (the Omni DiT is forced fp32 on GPU). +- **ExecuTorch**: model the runner on `examples/models/voxtral_tts/voxtral_tts_runner.cpp` (`AudioChunkCallback`, `streaming_chunk_frames_`, left context). Export three programs: talker (`export_llm` with a Qwen3 config; the code-predictor as a separate method), decoder (from `modeling_qwen3_tts_tokenizer_v2.py`, with explicit state tensors as method I/O in the static-attention style so QNN/CoreML can run it). Upper-bound dims: 25 + 72 frames for the decoder, 4096 for the talker. + +## 5. Parity and validation + +- Oracle: vLLM-Omni offline run of the same text/speaker with `temperature 0` (or a fixed seed) via `Omni.generate`, saving codec tokens and PCM; the session driver in [experiments/vllm_omni_tts/tts_stream_bench.py](experiments/vllm_omni_tts/tts_stream_bench.py) already records chunk boundaries. +- Metrics: token-level agreement of codebook 0 under greedy decoding (talker parity), PCM SNR of the decoder given identical codes (decoder parity), then perceptual checks (ASR WER on the 12 benchmark prompts, speaker similarity via the same ECAPA encoder). +- Streaming metrics identical to the server definitions in [experiments.md](experiments.md): TTFA, inter-chunk gap p95, RTF (streamed + tail), plus device-only metrics: resident memory, CPU/GPU utilization, and SoC temperature over a 10-minute continuous synthesis run. + +## 6. Blockers and unknowns + +- Decoder cost on ARM CPU/GPU is unmeasured; if a 25-frame window exceeds 2 s of compute on the target, use smaller steady chunks (the server supports `codec_chunk_ramp` and adaptive chunking, so the schedule is a parameter, not a redesign) [Verified server feature; edge cost Unknown]. +- llama.cpp and MNN Qwen3-TTS conversions come from their own converters (GGUF, `.mnn`); numerical parity with vLLM-Omni was not measured here. +- Voice cloning (Base variant) requires the ECAPA encoder and reference-audio codes on device; only speaker-embedding mode is confirmed in MNN [Verified]. +- Checkpoint/task parity is not established across engines: the server benchmark used CustomVoice 1.7B, llama.cpp's README demonstrates 1.7B Base, and MNN's demo requires reference-audio cloning and hard-codes `hiddenSize = 1024` in `Talker::generateQwen3TTS` (`omni.cpp`). Family support does not imply every size/task variant runs everywhere; Phase 0 must pin one checkpoint and task per engine before comparing. +- No edge hardware was available; every device number above is a projection to be replaced by Phase 0 measurements in [implementation_roadmap.md](implementation_roadmap.md). diff --git a/docs/analysis/vla_edge_design.md b/docs/analysis/vla_edge_design.md new file mode 100644 index 000000000..502a637db --- /dev/null +++ b/docs/analysis/vla_edge_design.md @@ -0,0 +1,131 @@ +# Edge VLA design (vLLM-Omni-compatible) + +Scope: run vision-language-action policies of the kind vLLM-Omni serves (GR00T N1.7, π0, and by extension InternVLA-A1) on edge devices, with vLLM-Omni's OpenPI endpoint as the behavioural oracle. Inference is separated from control and safety throughout; nothing here assumes hard real-time guarantees. Evidence: [engines/vllm_omni_models.md](engines/vllm_omni_models.md) §3, [engines/vllm_omni.md](engines/vllm_omni.md), the four runtime reports, and the CPU vision measurement in [experiments.md](experiments.md). + +## 1. Analysis of the pipelines being ported + +### 1.1 Perception + +| Model | Vision path (server) | Edge implication | +|---|---|---| +| GR00T N1.7 | Qwen3-VL encoder from Cosmos-Reason2-2B, backbone truncated to 12 layers, SDPA, eager, batch 1 [Verified] | ViT + 12-layer LLM per control step; exportable (HF `AutoModel` wrapper). | +| π0 | PaliGemma (SigLIP-So400m producing 256 × 1152 tokens per 224² image, per `modeling_pi0.py`; + Gemma-2B) with up to 3 cameras, prompt ≤ 48 tokens, fp32 [Verified] | Three So400m encoder passes per step. The CPU measurement (0.6–1.2 s per 512 px tile of SmolVLM's smaller SigLIP encoder at 8 x86 threads in llama.cpp) shows only that vision encoding dominates on CPU; it is not an estimate for π0's encoder, which must be measured from the actual graph. | +| DreamZero | 14B DiT world model, 16 UniPC steps, 3 cameras 352×640, per-session paged KV of 603 MiB [Verified] | Not an edge candidate at this size; kept as a cloud tier only. | + +### 1.2 Structured inputs and fusion + +Proprioception enters GR00T through a per-embodiment `CategorySpecificMLP` (32 embodiments, padded to 132 dims) and π0 through its state projection; language is tokenized once per task; fusion happens in the backbone (GR00T: `AlternateVLDiT` cross-attends to vision/text tokens; π0: Gemma expert attends to PaliGemma KV) [Verified]. In vLLM-Omni the observation dict travels in `OmniDiffusionSamplingParams.extra_args["robot_obs"]` set by `entrypoints/openpi/serving.py`, and never through `OmniPayload` [Verified]. Edge design: the observation struct is defined by the application and passed as plain tensors; there is no connector layer to preserve. + +### 1.3 Action generation and chunking + +| Model | Head | Steps | Horizon × dim | Output | +|---|---|---|---|---| +| GR00T N1.7 | `AlternateVLDiT` 16 blocks × 1536 | 4 Euler | 40 × per-embodiment (DROID: eef_9d 9, gripper 1, joint 7) | dict of action tensors [Verified] | +| π0 | Gemma-300M expert, flow matching | 10 Euler | 50 × 32 | (50, 32) [Verified] | +| InternVLA-A1 | flow head | 10 | 50 × 32 | offline only [Verified] | + +All are **stateless across inferences**: one observation → one action chunk; no KV reuse across control steps in the served code (eager, batch 1) [Verified]. Within one inference π0's `sample_actions` builds a prefix KV that the denoise loop reuses and draws Gaussian noise unless `noise` is passed [Verified]. For edge this means no persistent KV between steps, only weights and per-inference scratch, and that parity tests must pass the noise explicitly to make a step a deterministic function of the observation. + +### 1.4 Temporal state, latency, jitter, stale inputs + +- Temporal state lives in the **controller**, not the model: the action chunk (40–50 steps) is executed over several control ticks while the next inference runs [Inferred from the OpenPI one-request/one-chunk protocol]. The edge design keeps an explicit chunk buffer with timestamps. +- Latency per inference = preprocessing + encoder(s) + backbone + K flow steps. On the server there is no published per-step number in tree; InternVLA docs mention 38 s for its offline example (unmeasured claim) [Unknown]. +- Jitter sources on device: thermal throttling, GPU/NPU queue contention with the camera pipeline, garbage collection if any managed runtime is involved, and dynamic shapes (never allow them in the hot path). +- Stale inputs: the observation must carry a capture timestamp; if inference finishes later than a configured staleness bound, the controller keeps executing the previous chunk's remaining actions and the new chunk is time-aligned (skip the steps already elapsed) or discarded. + +### 1.5 Task success + +Success is measured in simulation first. vLLM-Omni ships simulation evaluation clients for DreamZero (`examples/online_serving/dreamzero/droid_sim_eval_client.py`, `molmospace_dreamzero_eval_demo.py`) that drive an external simulator through the server [Verified]; no simulator itself is in any of the six trees. The plan reuses those clients where the policy matches, still requires an external simulator, and defines success only relative to the server policy running the same episodes. + +## 2. Design + +### 2.1 Architecture: inference isolated from control and safety + +```mermaid +graph TB + subgraph sense["Sensors"] + CAM["Cameras (1–3)"] --> PRE + PROP["Joint/eef state"] --> OBS + end + PRE["Preprocess: fixed resize/normalize
(one tile per camera)"] --> OBS["Observation struct
images, state, task tokens, t_capture"] + OBS --> INF + subgraph inf["Inference process (best-effort, no RT guarantees)"] + INF["VLA edge runner
encoder → backbone → K-step flow head"] + INF --> CHUNK["Action chunk (H × D) + t_capture"] + end + CHUNK --> CTRL + subgraph ctrl["Control process (fixed rate, higher priority)"] + CTRL["Chunk buffer + time alignment
stale check, blending"] --> SAFE["Safety layer
limits, watchdog, stop"] + SAFE --> ACT["Actuators"] + end + CLOUD["vLLM-Omni OpenPI server
(parity oracle / re-planning tier, never in loop)"] -.-> CTRL +``` + +Rules [Proposal]: + +- Inference and control are separate OS processes (or at least threads with different priorities) sharing a lock-free single-slot mailbox for the latest chunk; the control loop never blocks on inference. +- Safety (joint limits, velocity limits, watchdog on chunk age, e-stop) is enforced after the controller and does not depend on the model runtime. +- The cloud endpoint is used only for parity tests and optional slow re-planning (task text updates), never for per-tick actions. + +### 2.2 Adapter interfaces [Proposal] + +```cpp +struct IVisionAdapter { virtual bool encode(const ImageTensor& img, int cam, TokenTensor& out) = 0; }; // fixed shape per camera +struct IFusionAdapter { virtual bool fuse(const TokenTensor* vis, int n_cam, const StateVec& s, const TaskTokens& t, Cond& out) = 0; }; +struct IActionHead { virtual bool sample(const Cond& c, int k_steps, ActionChunk& out) = 0; }; // K Euler steps inside +struct IVlaRunner { // composes the three, owns scratch + virtual bool infer(const Observation& o, ActionChunk& out, InferStats& st) = 0; // returns wall time, per-stage ms +}; +``` + +`infer` is stateless across calls; `Observation` carries `t_capture`. The runner may keep encoder outputs for cameras whose frames did not change (frame-hash check) as an optimization, which is safe because the model is stateless [Inferred]. + +### 2.3 State and buffer ownership + +| State | Owner | Notes | +|---|---|---| +| Weights, runtime scratch | Runner | Static allocation at load (ExecuTorch plans it at export; MNN/llama.cpp reserve at init). | +| Observation buffers | Sensor process | Double-buffered; the runner reads the latest complete frame. | +| Action chunk buffer | Controller | Ring of two chunks with timestamps; blending policy at chunk boundaries is the controller's. | +| Task text tokens | Application | Re-tokenized only when the instruction changes. | +| Per-embodiment tables (GR00T) | Runner | Constant tensors selected by embodiment id at load. | + +### 2.4 Stage placement per runtime [Proposal] + +| Runtime | Encoder | Backbone | Action head | Assessment | +|---|---|---|---|---| +| **ExecuTorch (primary)** | `vision_encoder` method (CoreML/QNN/XNNPACK) with fixed 224² input | `text_decoder`-style method with static shapes (prompt ≤ 48 tokens, no KV persistence) | Flow loop as a `while_loop`/unrolled K steps inside one method, or K calls from the runner | Static memory bound and zero-copy I/O are available mechanisms, not automatic: outputs must be left unplanned at export for `set_output_data_ptr` to accept them, and every op must be covered by the chosen delegate or it falls to portable CPU kernels; PT2E quantizer recipes per delegate; runner pattern exists in `MultimodalRunner`. Export sources: GR00T `diffusion/models/gr00t/` (HF wrapper), π0 `diffusion/models/pi0/modeling_pi0.py` (plain torch, parity test already runs on CPU [Verified]). | +| **MNN (secondary)** | `utils/vision.py` exporters (Qwen3-VL/SigLIP families) | `llmexport.py` path with `input_embeds` overloads, or ONNX | DiT via the diffusion engine or ONNX with a host loop | Broadest Android NPU coverage (QNN/Hexagon/NNAPI) and existing apps; batch 1 and static shapes are already the norm. | +| **llama.cpp** | `clip.cpp` projector families (SigLIP present) | `libllama` with `llama_batch.embd` for proprio/vision embeddings | Not available; would be a new C++ graph (sidecar proposal in [engines/llama_cpp.md](engines/llama_cpp.md)) | Viable for VLM-only reasoning tiers; not for the action head without new engine code. | +| **ncnn** | ViT/CNN encoder via pnnx | No | Small MLP head only | Component runtime for Vulkan-only devices. | + +Hardware placement: keep the encoder on the NPU/GPU and the flow head wherever K small matmuls run with the least launch overhead (often CPU for a 300 M expert at K = 4–10 [Inferred]); measure both. + +### 2.5 Latency budget template [Proposal] + +For a target control period T (for example 100 ms at 10 Hz): + +| Stage | Budget | How enforced | +|---|---|---| +| Preprocess | ≤ 5 % T | Fixed resize on the ISP/GPU | +| Encoders (n cameras) | ≤ 40 % T | One tile per camera; NPU delegate; skip unchanged frames | +| Backbone + fusion | ≤ 30 % T | Static shapes; quantized weights | +| Action head (K steps) | ≤ 20 % T | K fixed; unrolled | +| Slack for jitter | ≥ 5 % T | Monitored p95; if exceeded for N consecutive steps, drop to a lower-rate mode | + +Chunking makes the effective requirement softer: with a 40-step chunk executed at 10 Hz, inference may take up to several periods as long as chunk age stays within the staleness bound; the design treats T as the *desired* inference cadence, not a deadline. + +## 3. Validation in simulation first + +1. **Numerical parity** (no robot): export each module, feed recorded observations from an episode, compare action chunks with vLLM-Omni's OpenPI server (`examples/online_serving/pi0/openpi_client.py`) at fixed seeds; report max-abs and per-dimension RMS error per flow step count. +2. **Closed-loop in simulation**: the same episodes in a simulator with the edge runner in the loop; success rate versus the server policy on ≥ 50 episodes per task; report jitter (p95 of inference wall time) and chunk-age distribution. +3. **Fault injection**: delayed frames, dropped cameras, thermal throttling emulation (CPU frequency cap); verify the controller's stale-input path and the safety layer engage as specified. +4. Only after 1–3 pass on the target device class does hardware-in-the-loop start, with the safety layer active and independent of the model runtime. + +## 4. Blockers and unknowns + +- No VLA model runs in any of the four runtimes today; every path is an export project [Verified absence]. +- Model sizes: GR00T ≈ 6 GiB BF16 per recipe; π0 backbone is a 2B PaliGemma; both need weight quantization to fit under 4 GB on phones, and the quality impact on actions is unmeasured [Unknown]. +- Vision cost on CPU is the dominant risk (measured on x86 with llama.cpp: 0.6–1.2 s per 512 px SigLIP tile at 8 threads); NPU delegation of the encoder is required for anything above 1–2 Hz on CPU-only boards [Inferred]. +- Weights for DreamZero/π0/GR00T are not in the local cache and their download requires permission; reference numbers on the L20X could not be produced in this session. +- Simulation evaluation clients exist in vLLM-Omni (`examples/online_serving/dreamzero/droid_sim_eval_client.py`, `molmospace_dreamzero_eval_demo.py`), but the simulator itself and any π0/GR00T-specific harness remain external dependencies. diff --git a/docs/analysis/vllm_omni_edge_feasibility.md b/docs/analysis/vllm_omni_edge_feasibility.md new file mode 100644 index 000000000..a1bc3e754 --- /dev/null +++ b/docs/analysis/vllm_omni_edge_feasibility.md @@ -0,0 +1,130 @@ +# Feasibility: extending vLLM-Omni to edge devices + +Question answered here: can vLLM-Omni's TTS and VLA pipelines run on edge devices, and if so, which pieces of vLLM-Omni are re-used, replaced, or re-implemented? Evidence comes from [engines/vllm_omni.md](engines/vllm_omni.md), [engines/vllm_omni_models.md](engines/vllm_omni_models.md), [engines/vllm.md](engines/vllm.md), the four runtime reports, and [experiments.md](experiments.md). Labels as in [README.md](README.md). + +## 1. What "vLLM-Omni on edge" can mean + +vLLM-Omni is three things at once [Verified]: + +1. **A process/serving model**: `AsyncOmniEngine` → `Orchestrator` → per-stage vLLM engine processes (`StageEngineCoreProc`) or diffusion workers, wired by connectors (default: host-staged shared memory), exposed over HTTP/WebSocket. Initialization observed at 157–188 s for a 2-stage TTS on an L20X in the default configuration; the minimum with `enforce_eager`/parallel stage init was not measured. +2. **Pipeline semantics**: stage graph per model (`PipelineConfig`), deployment overlay (`DeployConfig`), async-chunk hand-off (1-frame first chunk, 25-frame steady chunks, 72-frame decoder context for Qwen3-TTS), abort propagation, output templates, OpenPI robot protocol. +3. **Model code**: talkers built on vLLM layers; codecs/vocoders/action heads that are plain PyTorch modules. + +Only (2) and parts of (3) are portable. The plan below therefore distinguishes: *re-host* (run the real vLLM-Omni on an edge server), *substitute* (keep vLLM-Omni's control plane, swap execution), *orchestrate lightly* (re-implement the control plane in a small runtime), *split* (edge/cloud hybrid), and *edge-native baseline* (use a runtime's own pipeline and compare). + +## 2. Target hardware classes and assumptions + +No edge hardware was available in this session (x86 host only). Statements about targets are drawn from what each repository ships for that target and are labelled. + +| Target | Assumption made here | vLLM / vLLM-Omni status | Runtime coverage | +|---|---|---|---| +| **Embedded NVIDIA (Jetson Orin / Thor class)** | aarch64 Linux, CUDA, 8–64 GB unified memory, JetPack; power caps 15–60 W | vLLM: Tegra detection path and CUDA arch 8.7/11.0 in the build list, no wheels/CI/docs beyond an sm_110a note [Verified]. vLLM-Omni: CUDA platform code is arch-agnostic; aarch64+CUDA is exercised in vLLM CI on GH200 only [Verified]. **Feasible in principle, unproven** [Inferred]. | llama.cpp CUDA (needs explicit sm_87), MNN CUDA (sm87 in list), ExecuTorch CUDA (AOTInductor; Jetson unknown), ncnn (aarch64 CPU + Tegra Vulkan) [Verified] | +| **ARM CPU (Linux SBC, Graviton-class, phone big cores)** | ARMv8.2+ with fp16/dotprod, often i8mm; 4–8 usable cores; 4–16 GB | vLLM CPU platform is first-class on aarch64 Linux (wheels, ARM CI, oneDNN+ACL, KleidiAI W4A8) [Verified]; vLLM-Omni **has no CPU platform** [Verified]. | All four runtimes; KleidiAI in llama.cpp, MNN, ExecuTorch(XNNPACK) [Verified] | +| **Apple Silicon (M-series Mac, iPhone/iPad A-series)** | macOS/iOS, Metal, ANE via CoreML, unified memory | vLLM: CPU-only experimental on macOS; Metal via out-of-tree `vllm-metal` (MLX) [Verified]; vLLM-Omni: nothing [Verified]. **Not feasible as-is** on iOS (no Python/torch runtime model). | llama.cpp Metal + xcframework; MNN Metal/CoreML + iOS framework; ExecuTorch CoreML/MLX/Metal via SwiftPM; ncnn via MoltenVK [Verified] | +| **Mobile GPU/NPU (Android: Adreno/Mali GPU, Hexagon/QNN, MediaTek, Samsung)** | Android 10+, 8–16 GB, thermal throttling within minutes | vLLM/vLLM-Omni: nothing [Verified]. **Not feasible.** | llama.cpp OpenCL (Adreno), Vulkan, Hexagon HTP; MNN OpenCL/Vulkan/QNN/Hexagon/NNAPI; ExecuTorch Vulkan/QNN/MediaTek/Samsung; ncnn Vulkan [Verified] | + +## 3. Strategy comparison + +```mermaid +graph LR + A[Cloud / edge-server tier
vLLM-Omni as-is] --> B{Strategy} + B --> S1["S1 Direct port
run vLLM-Omni on Jetson/ARM"] + B --> S2["S2 Backend substitution
keep Orchestrator, swap stage workers"] + B --> S3["S3 Lightweight orchestration
re-implement stage graph in C++ on a runtime"] + B --> S4["S4 Hybrid edge/cloud
split stages across the network"] + B --> S5["S5 Edge-native baseline
runtime's own pipeline, compare"] +``` + +| Strategy | What it keeps from vLLM-Omni | Targets it can reach | Blocking facts | Verdict | +|---|---|---|---|---| +| **S1 Direct port** (install vLLM + vLLM-Omni on the device) | Everything | Jetson-class CUDA only (aarch64 Linux + CUDA); ARM CPU only if a CPU platform is added to vLLM-Omni | Initialization 157–188 s observed on L20X with default compile/graph capture, likely worse on Jetson [Inferred]; Python + torch 2.13 + transformers + diffusers footprint; two engine processes for two AR/generation stages; no CPU platform in vLLM-Omni (missing implementation, not architectural) [Verified] | Viable for an **edge server** (Orin 64 GB, Thor) where long-lived processes are fine; not viable for phones, MCUs, or cold-start use. Recommended as the *reference tier* only. | +| **S2 Backend substitution** (vLLM platform plugin whose worker calls llama.cpp/MNN/ExecuTorch) | Orchestrator, schedulers, connectors, APIs | Same as S1 (still Python + vLLM process model) | The vLLM platform plugin contract requires a `WorkerBase`, an `AttentionBackend`, a device communicator and torch tensors flowing through the model runner [Verified]; a worker that delegates a whole stage to a foreign runtime bypasses paged attention, sampling and CUDA-graph logic, so most of what vLLM contributes is lost while its process overhead stays. `OmniPayload` has no action keys; the default shared-memory connector serializes via CPU; the Mooncake/Mori transfer-engine connectors have device fast paths (Mori's `xgmi` backend is intra-node), but none is validated for an edge target [Verified]. | Not recommended as the main path. A narrow variant is useful: **replace only the vocoder/codec stage** with an exported artifact executed by torch (or ExecuTorch's Python runtime) inside the existing `LLM_GENERATION` worker on Jetson. | +| **S3 Lightweight orchestration** (re-implement `PipelineConfig` semantics in a small C++ runtime on top of llama.cpp / MNN / ExecuTorch) | Stage graph, chunk schedule, state-ownership rules, abort semantics, API shapes, tests as parity oracle | All four target classes | Requires exporting every stage to the runtime's format; talkers are vLLM-bound modules and must be re-exported from the HF checkpoint, not from vLLM-Omni code [Verified]; no shared connector abstraction exists across the runtimes (all in-process, app-owned buffers) | **Recommended primary path** for phones, Apple devices, ARM boards. Two of the runtimes already implement the Qwen3-TTS graph natively (llama.cpp `MTMD_GEN_AUDIO_TYPE_QWEN3TTS`, MNN `Llm::generateTTS`), so the TTS orchestration layer mostly adds streaming, interruption and parity, not model porting. | +| **S4 Hybrid edge/cloud** (e.g. thinker or talker in the cloud, codec/vocoder on device; or perception on device, policy in cloud) | Cloud stages unchanged; device runs one exported stage | All targets for the device stage | vLLM-Omni's connectors are process/host transports (shared memory, RDMA) not WAN-ready [Verified]; the WebSocket endpoints (`/v1/audio/speech/stream`, `/v1/realtime`, OpenPI) are the practical boundary. Network jitter dominates for VLA; for TTS a codec-token stream is tiny (16 codebooks × 12.5 Hz) which makes talker-in-cloud / vocoder-on-device attractive [Inferred]. | Recommended as the **first deliverable** for TTS (codec tokens over WebSocket, on-device vocoder), **conditional on** a prototype of the new wire contract (the speech stream endpoint's `_generate_and_send` consumes PCM today, so codec-token streaming needs stage termination, token ordering, decoder state/revision, EOS and cancel semantics defined) and on a measured on-device decoder; and as a fallback tier for VLA (never in the control loop, only for re-planning). | +| **S5 Edge-native baseline** (llama.cpp `llama-tts`, MNN `qwen3_tts_demo`, ExecuTorch Voxtral runner; VLM via `llama-mtmd-cli`) | Nothing but the model weights and the benchmark protocol | All targets | These paths are single-request, mostly non-streaming (llama.cpp CLI, MNN Qwen3-TTS single callback) [Verified] | Required as the **measurement floor** in every phase; also the fastest way to get audio on a device today. | + +Assessment of the "must keep vLLM-Omni" premise: an edge deployment that keeps the vLLM-Omni *code* is only sensible on Jetson-class servers (S1). For every other target, the deliverable is "vLLM-Omni-compatible edge pipelines": same models, same chunking and state rules, same API surface, validated against vLLM-Omni outputs. That is S3 with S4 and S5 as bookends. + +## 4. Per-runtime assessment for stage placement + +Stages considered (from [engines/vllm_omni_models.md](engines/vllm_omni_models.md)): text front-end + speaker conditioning (ECAPA), **AR talker** (Qwen3 LLM with 15/16-codebook code predictor), **codec decoder / vocoder** (RVQ decode + sliding-window transformer + BigVGAN), and for VLA: **vision encoder** (SigLIP/Qwen3-VL ViT), **state/language fusion** (LLM backbone), **action head** (flow/diffusion DiT, 4–10 Euler steps). + +### 4.1 llama.cpp + +| Aspect | Finding | +|---|---| +| Stage placement | Talker + code predictor + code2wav + speaker encoder all inside `libmtmd` for Qwen3-TTS (`tools/mtmd/models/qwen3tts-gen.cpp`, `qwen3tts-spkenc.cpp`; enum in `tools/mtmd/mtmd.h`) [Verified]. Vision encoder for a VLM in `clip.cpp` with ~60 projector types; LLM backbone in `libllama`; **no action head** [Verified]. | +| Operator gaps | For TTS: none for Qwen3-TTS and Pocket-TTS; other vLLM-Omni families (CosyVoice3 CFM+HiFT, Fish DAC, MOSS codec) would need new graph code in C++ [Verified absence]. For VLA: flow head = new graph; proprio input has to enter through `llama_batch.embd`; `while`-style denoise loop must be driven from the app. `ggml` has conv/pool/1-D ops and flash attention, so the DiT itself is expressible [Inferred]. | +| State | KV in `llama_kv_cache` (per-seq, ring, quantizable, save/load); code2wav 72-frame window state inside the mtmd helper; caller-owned state blobs in the step API [Verified]. | +| Copies | Zero-copy mmap weights; scheduler copies at backend splits; audio helper returns float buffers [Verified]. | +| Integration effort | TTS: low (already runs); work is streaming callback + interruption + parity harness. VLA: medium-high (C++ graph for the action head + app loop). | +| Where it fits | ARM CPU (KleidiAI), Apple Metal, Android via OpenCL/Adreno, Hexagon HTP, Jetson CUDA; static single binary. | + +### 4.2 ncnn + +| Aspect | Finding | +|---|---| +| Stage placement | Vocoder/VITS-class nets and CNN/ViT vision encoders; the Piper example shows a 5-net app-driven TTS [Verified]. LLM talker only via the batch-1 KV cache path with host-provided masks and RoPE, no tokenizer, no sampler, seq-len-1 export [Verified]. | +| Operator gaps | No Vulkan for Embed/MatMul/GLU/RNN; no tokenizer; no audio I/O; block-quant LLM only on CPU [Verified]. | +| State | App-owned `Mat` KV blobs (consume-and-replace); no paging [Verified]. | +| Copies | Explicit upload/download per extract on Vulkan [Verified]. | +| Integration effort | High for anything LLM-shaped; low for a stand-alone vocoder or vision encoder. | +| Where it fits | Only as a **component runtime** (vocoder, image encoder, small MLP/CNN action head) inside an S3 orchestrator on Vulkan-only devices; not recommended as the primary TTS/VLA runtime. | + +### 4.3 MNN + +| Aspect | Finding | +|---|---| +| Stage placement | Full Qwen3-TTS path in the LLM engine (talker → 16-group code predictor → codec embedder → speech decoder; `Llm::generateTTS`, `demo/qwen3_tts_demo.cpp`) and Qwen2.5-Omni talker → DiT → BigVGAN with chunked callback [Verified]. VLA: vision exporters for 16 encoder families, DiT/flow runtime in the diffusion engine, `input_embeds` overloads; no policy model [Verified]. | +| Operator gaps | Qwen3-TTS decoder runs as **one call per utterance** (single callback) rather than chunk-streamed; GPU backends ignore `attention_mode` (KV quant/flash only on CPU/Metal) [Verified]. Action head: DiT ops exist; proprio MLP + embodiment lookup is ordinary graph code [Inferred]. | +| State | `KVMeta` per forward, contiguous KV with quant and mmap-to-disk; codec state in engine worker threads; `Module::clone` shares weights [Verified]. | +| Copies | Per-op CPU fallback wraps; NPU subgraph compile is all-or-nothing [Verified]. | +| Integration effort | TTS: low-medium (add chunked streaming to the Qwen3-TTS path, mirror vLLM-Omni's 1/25/72 schedule). VLA: medium (export encoder + backbone + head via `llmexport.py`/ONNX, write the control loop in C++ against `Module`). | +| Where it fits | Android (CPU/OpenCL/QNN/Hexagon/NNAPI), iOS/macOS (Metal/CoreML), OpenHarmony, Jetson CUDA; the only runtime with in-tree Android/iOS chat apps and an Omni engine. | + +### 4.4 ExecuTorch + +| Aspect | Finding | +|---|---| +| Stage placement | LLM runner (`TextLLMRunner`, `MultimodalRunner`) for talker/backbone; Voxtral TTS example shows LM + flow head + codec decoder as two `.pte`s with a chunked streaming callback (25 frames, 5 initial, 25 left context) [Verified]. VLA: any sub-module exportable with `torch.export`; `while_loop` supported inside graphs; static planning and zero-copy I/O suit fixed-rate loops [Verified]. | +| Operator gaps | Per delegate: QNN/Ethos-U static shapes only; CoreML/XNNPACK coverage decided by partitioner; unsupported ops fall to portable CPU kernels, which is correct but slow [Verified]. The Qwen3-TTS decoder's Snake activations and sliding-window attention have no delegate-specific kernels [Unknown until exported]. | +| State | KV as in-graph mutable buffers with upper-bound length (or static-attention I/O for NPUs); runner tracks position; no paging [Verified]. | +| Copies | Inputs aliased; outputs zero-copy only when the export leaves them unplanned (`Method::set_output_data_ptr` rejects memory-planned outputs; see `docs/source/compiler-memory-planning.md`); `_h2d/_d2h` only on CUDA delegates [Verified]. | +| Integration effort | TTS: medium (export talker + code predictor + decoder; write a Qwen3-TTS runner modelled on the Voxtral runner). VLA: medium (export three modules; runner loop; QNN/CoreML quantizer recipes). Highest AOT rigor (memory bound known at build time), which is what a safety-reviewed control path wants. | +| Where it fits | Apple (CoreML/MLX), Qualcomm QNN, MediaTek/Samsung NPUs, XNNPACK ARM CPU, Cortex-M/Ethos-U for tiny heads; CUDA on Linux (Jetson unknown). | + +## 5. Source-level integration points in vLLM-Omni + +These are the places an edge implementation reads from (as specification) or hooks into (for hybrid/parity), all [Verified] in the engine reports: + +| Purpose | Path | Use | +|---|---|---| +| Stage graph per model | `vllm_omni/model_executor/models//pipeline.py`, registry `vllm_omni/config/pipeline_registry.py` | Source of truth for stage order, types, and output modality; the edge orchestrator mirrors it | +| Chunk schedule and decoder context | [`vllm_omni/deploy/qwen3_tts.yaml`](../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml) (`initial_codec_chunk_frames`, `codec_chunk_frames`, `codec_left_context_frames`), `chunk_size_utils.py` | Same numbers on device so audio is bit-comparable | +| Payload contract | `vllm_omni/data_entry_keys.py` (`OmniPayload`: hidden_states, embed, ids, codes, meta) | Defines what crosses a stage boundary; edge adapters carry the same keys | +| Streaming/abort semantics | `Orchestrator._handle_abort`, `OmniChunkTransferAdapter`, `WAITING_FOR_CHUNK` | Interruption rules to replicate | +| TTS output formatting | `vllm_omni/model_executor/models/output_templates.py`, TTS adapter registry | Output shape (`audio`, `sr`, chunk list) the edge API should match | +| Robot API | `vllm_omni/entrypoints/openpi/` (`serving.py` puts observations into `robot_obs`), `diffusion/output_formatter.py` (`actions`) | Wire protocol for hybrid VLA and for parity tests | +| Parity oracles | `tests/e2e/offline_inference/test_qwen3_tts_*.py`, `tests/e2e/online_serving/test_{pi0,gr00t_openpi,dreamzero}_expansion.py`, CPU unit tests per family | Reference outputs for edge validation | +| Model code that is plain torch | Qwen3-TTS `tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py` (decoder), `common/qwen3_code_predictor.py`, GR00T `diffusion/models/gr00t/`, π0 `diffusion/models/pi0/` | Export sources for ExecuTorch/MNN/ONNX | +| Model code that is vLLM-bound | Talker classes using `Qwen2Model`/`Qwen3Model`, `ParallelLMHead` | Must be re-exported from the HF checkpoint, not from these classes | + +## 6. Blockers + +| Blocker | Type | Affects | +|---|---|---| +| No CPU platform in vLLM-Omni | Missing implementation | S1/S2 on ARM CPU | +| Initialization of 2.5–3 min for two stages (observed, default config; lower bound unmeasured) | Process model is architectural; compile/graph capture at start is configurable (`enforce_eager`, `compilation_config`) | S1 on any device with cold-start needs | +| Default inter-stage transport (`SharedMemoryConnector`) is host-staged; device fast paths exist in the Mooncake/Mori transfer-engine connectors (including Mori's intra-node `xgmi` backend on AMD), none validated for an edge target | Missing validation/implementation for edge co-location (repo docs list D2D as roadmap) | S1/S2 latency; irrelevant to S3 | +| Talkers depend on vLLM layers | Architectural for reuse; not for re-export | S3 must export from HF checkpoints | +| `OmniPayload` lacks action/video keys; VLA bypasses connectors | Missing implementation | S2/S4 for VLA | +| Qwen3-TTS decoder streaming in MNN is single-callback; llama.cpp CLI non-streaming | Missing implementation | S3 on MNN/llama.cpp | +| No VLA model in any runtime | Missing implementation | S3/S5 for VLA | +| No edge hardware in this environment | Environmental | All measurements beyond x86/L20X | +| Model weights for VLA (DreamZero, π0, GR00T) not cached; downloads need permission | Environmental | Reference measurements for VLA | + +## 7. Decision + +- **TTS**: S4 first (talker in vLLM-Omni, on-device vocoder) **only if** the codec-token wire-contract prototype and the on-device decoder measurement pass (the speech stream endpoint's `_generate_and_send` consumes PCM today); otherwise S3 first. Then S3 on llama.cpp (ARM CPU, Apple) and MNN (Android NPU/GPU, iOS) with ExecuTorch for CoreML/QNN targets, S5 numbers reported alongside at every phase. Design in [tts_edge_design.md](tts_edge_design.md). +- **VLA**: S3 only, on ExecuTorch (primary, for its static memory model) and MNN (secondary, Android/NPU breadth), with vLLM-Omni's OpenPI endpoint as the parity oracle and as an S4 re-planning tier that is never inside the control loop. Design in [vla_edge_design.md](vla_edge_design.md). +- **S1** is retained for Jetson-class edge servers as the way to run the unmodified reference and to validate the other tiers. diff --git a/docs/edge/comparisons.md b/docs/edge/comparisons.md new file mode 100644 index 000000000..a48b0cbc5 --- /dev/null +++ b/docs/edge/comparisons.md @@ -0,0 +1,95 @@ +# Baselines and cross-engine comparison + +Where this fork's numbers sit against other engines, and what each comparison +does and does not establish. + +## The baseline engine: llama.cpp + +llama.cpp is the reference for CPU inference of a quantized model, so it is what +"good" is measured against. Both engines are pinned to the same physical cores +with `taskset` and run **strictly one at a time**; vLLM's prefix caching is +disabled so a replayed prompt is really recomputed, because `llama-bench` has no +such cache and leaving it on would not be a comparison. + +Spark-X2.5-1.7B, Xeon Platinum 8480C: + +| | decode (tg128) | prefill (pp512) | +|---|---|---| +| llama.cpp Q4_K_M | **90.0 tok/s** | 1121 tok/s | +| this fork, W4A8 AMX | 74.4 tok/s | — | +| this fork, W4A16 tinygemm | 71.6 tok/s | **2190 tok/s** | + +**Decode is llama.cpp's.** Prefill is ours by ~2×, which is the shape one expects: +vLLM's prefill goes through oneDNN/AMX bf16 GEMMs at large batch, where +llama.cpp's advantage in per-step overhead does not apply. + +Not a matched-precision comparison in the strict sense: Q4_K_M stores **4.5 +bits/weight** over groups of 32 with higher-bit types for some tensors, against +our **4.25** over groups of 128. It buys fidelity with those bits (below). + +### Why llama.cpp wins decode + +Both engines are far from the hardware limit. A node-local streaming read on the +same 24 cores sustains 222 GB/s: + +| | GB/s | of ceiling | +|---|---|---| +| llama.cpp whole step | 86 | 39% | +| this fork, whole step | 73 | 33% | + +So the gap is not bandwidth, and it is not primarily kernel quality — it is the +execution model. `ggml_threadpool` starts threads once and they walk the entire +graph together, synchronising at `ggml_barrier` through an atomic spin. vLLM +enters a separate `omp parallel` region inside each kernel, and a region costs +~1.2 µs at 24 threads. See [ROADMAP.md](../../ROADMAP.md) item I10 for the +measurements that size this and the reason it is not the next thing to build. + +### Fidelity + +Against each build's own bf16 output, 12 greedy prompts, exact reproductions: + +| | | bits/weight | +|---|---|---| +| llama.cpp Q4_K_M | 4/12 | 4.5 | +| W4A16 | 2/12 | 4.25 | +| W4A8 | 1/12 | 4.25 | + +Some of that gap is simply the extra quarter-bit and the finer grouping. How +much is unknown, which is why closing the budget gap (g64, or fp16 scale+zero — +both config changes) comes before implementing a better quantizer. + +This metric also saturates: it discards everything after the first divergent +token. `benchmarks/edge_harness/fidelity_kl.py` replaces it with top-1 agreement +and KL over a corpus, deliberately symmetric with `llama-perplexity +--kl-divergence` so llama.cpp stays comparable on our metric rather than being +judged on a different one. + +## Other engines considered + +Full analyses in [docs/engines/](../engines/). In brief, for this workload: + +| engine | verdict | +|---|---| +| **llama.cpp** | the CPU baseline; best decode, mature quantization, no multimodal pipeline | +| **ExecuTorch** | the phone path — exported artifacts, XNNPACK on CPU, QNN for NPU; not a server engine | +| **MNN** | strong mobile CPU/GPU, own model format, another export toolchain | +| **ncnn** | mobile-first, no LLM serving story of the kind needed here | +| **vLLM / vLLM-Omni** | the only one with a multi-stage multimodal pipeline, continuous batching and an OpenAI-compatible server — which is why the edge work is a fork of it rather than a port away from it | + +The decision recorded in [docs/analysis/comparison.md](../analysis/comparison.md) +is that a phone NPU cannot run vLLM at all: there the deliverable is an exported +graph, and the quantization scheme is decided by the export toolchain. On device +(Snapdragon via QNN) only calibrated **w8a16** was measured correct — 36.9 dB; +int8 activations break it at 5.1 dB, and uncalibrated w4a16 scores −1.6 dB, i.e. +uncorrelated with the reference. + +## What the comparisons do not establish + +- **ARM.** Every number here is x86 with AMX. `cpu_gemm_wna16` was measured on + its AMX path because `_get_isa_hint` returns `"amx"` whenever AMX is present; + the `"vec"` path a genuinely AMX-less machine takes is untested. +- **Task-level quality.** One run per cell on the agent suite showed that + navigation survives quantization and recognising completion does not. That is + enough to show a failure reproduces, not to rank builds. +- **Other models.** Qwen3-TTS and MiniCPM-o have edge profiles and init work in + this tree, but the 4-bit CPU results above are Spark-X2.5 only. diff --git a/docs/edge/environment.md b/docs/edge/environment.md new file mode 100644 index 000000000..67561b236 --- /dev/null +++ b/docs/edge/environment.md @@ -0,0 +1,79 @@ +# Environment, pins and reproducing a number + +Every figure in this repository was produced with the versions below. A result +whose environment was not recorded cannot be retracted later, only doubted, so +the harness stamps these into each artifact automatically — see the +`provenance` block in any file under `benchmarks/edge_harness/results/`. + +## vLLM is **not** modified + +Worth stating plainly, because it is the obvious assumption and it is wrong. + +This fork changes vLLM's *behaviour* on CPU, but the vLLM tree and the installed +wheel are stock. The changes are runtime patches applied at import through +vLLM's own extension points, so they survive a wheel upgrade instead of being +lost on it: + +| patch (`vllm_omni/patch.py`) | what it changes | +|---|---| +| `_patch_cpu_aot_compile_cache_load` | vLLM's CPU path fails to load its own AOT artifact (`finalize_loading` missing on a plain function), treats it as a miss, and recompiles on every start | +| `_patch_cpu_explicit_kv_cache_memory` | `CpuPlatform.check_and_update_config` overwrites `kv_cache_memory_bytes` from `VLLM_CPU_KVCACHE_SPACE` unconditionally, so a per-stage budget had no effect at all — this is what made "KV sizing is not a memory lever" look true | +| `_patch_cpu_release_memory_after_load` | returns freed allocator arenas to the OS after weight load | + +Other extension points used, rather than edits: the quantization-config registry +(`register_quantization_config` for `cpu_int4`), the IR op registry, platform +hooks, and worker method wrapping. + +So there is no vLLM patch series to apply. Pin the version and let `patch.py` +do the rest. + +## Pins + +| component | pin | +|---|---| +| vLLM (installed) | `0.28.0+cpu` wheel | +| vLLM (source read for this work) | `51da0ca66c8065619c79e35dff97aa99aeaf5644` (`v0.28.1rc0`) | +| llama.cpp (baseline) | `e71b80510c848c00175924ecf3c40333ccae8eb5` (2026-09-07) | +| Python | 3.12 | +| torch | CPU build, as pulled by the vLLM CPU wheel | + +```bash +pip install --torch-backend cpu \ + https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0+cpu-cp38-abi3-manylinux_2_34_x86_64.whl +VLLM_TARGET_DEVICE=cpu pip install -e . # this fork +``` + +## Machine + +Dual-socket Xeon Platinum 8480C (Sapphire Rapids, 56 cores/socket, AMX-BF16 and +AMX-INT8), 2 TB RAM, NUMA nodes 0 = CPUs 0-55/112-167 and 1 = 56-111/168-223. + +It is **shared**, which shapes everything: end-to-end throughput swings ~30% run +to run, per-op self time reproduces to ~5%, `RssAnon` to 0.2%. Runs are pinned +with `numactl --physcpubind=<24 cores> --membind=` and the harness refuses +to start when those cores are already busy. + +## Reproducing a published number + +```bash +# memory: any of the four levers, as its own A/B +python benchmarks/edge_harness/bench_harness.py --spec .json \ + --passes 3 --metric rss_anon_tree_mb --cores 56-79 --out summary.json + +# where each number came from +python -c "import json,sys; print(json.load(open(sys.argv[1]))['provenance'])" \ + benchmarks/edge_harness/results/.json +``` + +Two settings change what a memory or profile number *means*, and both have +produced a wrong published conclusion by being left at their default: +`--context-len` (without it the KV cache is allocated and never touched) and +`--prompt-len` (the default of 1 profiles an empty attention cache). + +## Retracted artifacts + +Files under `benchmarks/edge_harness/results/` carrying `"retracted": true` are +kept deliberately rather than deleted, with the reason and the superseding file +named. `profile_w4a8_g128_p{1,2}.json` are the clearest case: both show +`_C::onednn_mm` at ~2.97 ms/step, the signature of a checkpoint whose output +head was left in bf16, and every W4A8 number derived from them is void. diff --git a/docs/engines/comparison.md b/docs/engines/comparison.md new file mode 100644 index 000000000..ee94f77d9 --- /dev/null +++ b/docs/engines/comparison.md @@ -0,0 +1,163 @@ +# Six-engine comparison + +Sources: the seven engine reports under [engines/](engines/) (each claim there carries a file/symbol citation) and the measurements in [experiments.md](experiments.md). This document compares; it does not re-derive. Labels: [Verified] = confirmed in the engine reports against source at the analyzed commits; [Inferred]; [Proposal]; [Unknown]; "repo-reported" = published by the repository, not reproduced; "measured" = produced in this session on this x86/L20X host. + +## 1. Two kinds of software + +The six repositories split into **serving stacks** (own a request queue, batch across requests, run as long-lived processes) and **execution runtimes** (a library that runs one graph or one decode step when called). This distinction explains most of the comparison below, and it is the first thing the edge plan has to respect: a serving stack can be *re-hosted*, but it cannot be *shrunk* into a runtime without replacing its process model. + +```mermaid +graph TB + subgraph serving["Serving stacks (request scheduler + process model + HTTP)"] + VO["vLLM-Omni
multi-stage orchestrator over vLLM engines"] + V["vLLM
V1 engine: scheduler, paged KV, executor"] + VO -->|"subclasses EngineCoreProc, Scheduler, model runners; monkeypatches ~10 vLLM symbols"| V + end + subgraph runtimes["Execution runtimes (library: load graph, run, return)"] + L["llama.cpp
ggml graph + libllama + libmtmd
(+ optional llama-server with slot scheduler)"] + M["MNN
graph engine + LLM/Omni engine
(+ single-Llm HTTP wrappers)"] + N["ncnn
graph engine only"] + E["ExecuTorch
AOT compiler + .pte interpreter + LLM runner
(+ experimental batching lib, llm_server example)"] + end + V -->|"torch, CUDA/ROCm/XPU/CPU kernels"| T["PyTorch 2.13 runtime"] + L --> HW["CPU / Metal / Vulkan / OpenCL / CUDA / Hexagon / CANN ..."] + M --> HW + N --> HW2["CPU / Vulkan"] + E --> HW3["delegates chosen at export: XNNPACK / CoreML / MLX / QNN / Vulkan / CUDA ..."] +``` + +| Engine | Kind | Unit of work the public API accepts | Long-lived process model | Report | +|---|---|---|---|---| +| vLLM | Serving stack (V1 engine) | A request (prompt + sampling params) into a continuous-batching scheduler | Yes: API server + `EngineCoreProc` + workers, ZMQ between them [Verified] | [engines/vllm.md](engines/vllm.md) | +| vLLM-Omni | Serving stack built on vLLM | A request that flows through a DAG of stages (AR / generation / diffusion), each stage a vLLM-style engine process | Yes: `AsyncOmniEngine` + `Orchestrator` + one `StageEngineCoreProc` per AR stage, shared-memory or RDMA connectors between stages [Verified] | [engines/vllm_omni.md](engines/vllm_omni.md), [engines/vllm_omni_models.md](engines/vllm_omni_models.md) | +| llama.cpp | Execution runtime (+ optional server tool) | `llama_decode(batch)` over one or more `seq_id`s; `mtmd` chunk encode; audio-gen step API | No in `libllama`; yes only in `tools/server` (slots, single worker queue) [Verified] | [engines/llama_cpp.md](engines/llama_cpp.md) | +| ncnn | Execution runtime | `Extractor::extract(blob)` on one `Net` | No [Verified] | [engines/ncnn.md](engines/ncnn.md) | +| MNN | Execution runtime (+ LLM/Omni engine) | `Module::onForward`, or `Llm::response/generate` for one conversation | No; `mls_server`/`mnncli` wrap one `Llm` under a mutex [Verified] | [engines/mnn.md](engines/mnn.md) | +| ExecuTorch | Execution runtime (AOT + interpreter) | `Method::execute()` on one loaded `.pte`; `IRunner::generate` for LLMs | No in core; experimental `extension/llm/batching` scheduler has no in-tree executor; `examples/llm_server` serializes execution [Verified] | [engines/executorch.md](engines/executorch.md) | + +## 2. Device layer + +| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch | +|---|---|---|---|---|---|---| +| In-tree accelerators | CUDA, ROCm, XPU, CPU (x86/aarch64/Apple CPU); TPU via out-of-tree shim; Jetson detected through `/etc/nv_tegra_release` with CUDA arch 8.7/11.0 in the build list [Verified] | CUDA, ROCm, Ascend NPU (incl. 310P), XPU, MUSA; **no CPU platform** (`UnspecifiedOmniPlatform` reports 0 devices) [Verified] | 18 backend dirs: cpu, cuda, hip, musa, metal, vulkan, opencl (Adreno), hexagon (HTP), cann, sycl, openvino, webgpu, blas, rpc, virtgpu, zdnn, zendnn, et (a vendor accelerator platform under `/opt/et`, not ExecuTorch) [Verified] | CPU (ARM/x86/RISC-V/MIPS/LoongArch) + Vulkan only; Apple via MoltenVK [Verified] | 16 backend dirs: cpu (arm82/KleidiAI/SME2/AVX), metal, opencl, vulkan, opengl, cuda (sm60-120), musa, tensorrt, coreml, nnapi, qnn, hiai, hexagon, rknn, neuropilot (stub) [Verified] | Delegates chosen at export: XNNPACK, Vulkan, WebGPU, QNN, MediaTek, Samsung ENN, Ethos-U/VGF, NXP, OpenVINO, CoreML, Metal (exp.), MLX (exp.), CUDA (AOTInductor), Cortex-M, Cadence [Verified] | +| Discovery / selection | Exactly one `Platform` activates from builtin detectors or the `vllm.platform_plugins` entry point; out-of-tree wins [Verified] | Same as vLLM plus `vllm_omni.platform_plugins`; devices assigned per stage in deploy YAML [Verified] | `ggml_backend_registry`, dynamic backend `.so` loading, registration order = scheduler priority, CPU last [Verified] | Vulkan device by `rough_score`; CPU ISA tables at load [Verified] | `Schedule::getAppropriateType` AUTO priority HIAI → CoreML/NNAPI → TRT → CUDA → OpenCL → Metal → Vulkan → CPU [Verified] | None at runtime: `BackendDelegate::Init` looks up a string id in a 16-slot static registry and checks `is_available()` [Verified] | +| Memory model | Paged KV blocks sized from `gpu_memory_utilization` after a profile run; torch caching allocator; CuMem sleep mode [Verified] | Inherits; adds per-stage `determine_available_memory`, stage co-location admission, diffusion offload, host-weight mmap sharing [Verified] | `ggml_gallocr` worst-case compute buffer reserved at context init; mmap weights; per-backend buffer types [Verified] | Pool allocators, optional mmap weights, Vulkan blob/weight/staging allocators [Verified] | Eager/defer arena allocators, mmap weights and feature maps, KV cache mmap-to-disk option [Verified] | Static planning at export (`non_const_buffer_sizes`), `HierarchicalAllocator` arenas, zero-copy input aliasing and output pointers [Verified] | +| Host/device transfers | torch copies; multimodal embeddings scattered on device; CPU offload/KV connectors [Verified] | Default `SharedMemoryConnector` path: **D2H → shared memory (msgpack) → H2D** for every hop; repo docs state connectors are D2H2D, but `MooncakeTransferEngineConnector.put` and the Mori connector pass tensors/`ManagedBuffer` without serialization and support device memory pools, Mori including an intra-node `xgmi` backend [Verified] | Scheduler inserts copies at backend splits; async `tensor_set/get`; RPC backend serializes over TCP [Verified] | Explicit `record_upload/download` per extract; CPU fallback downloads [Verified] | `WrapExecution` copies at per-op fallback boundaries; NPU delegates run whole subgraphs [Verified] | `_h2d_copy/_d2h_copy` inserted by `PropagateDevicePass` at delegate boundaries (CUDA) [Verified] | +| Sync / streams | CUDA streams, CUDA graphs, async scheduling overlaps CPU/GPU [Verified] | Adds copy-stream D2H snapshot, async output materialization [Verified] | One stream per backend; events; pipeline parallelism up to 4 copies for multi-GPU [Verified] | One command buffer per extract, `submit_and_wait` [Verified] | Single queue per GPU backend, commit-and-wait; no multi-stream [Verified] | Synchronous interpreter; optional per-method CUDA graph [Verified] | +| Portability envelope | Linux servers; aarch64 Linux CPU wheels; macOS CPU experimental; no Android [Verified] | Linux CUDA/ROCm/NPU/XPU/MUSA servers only [Verified] | Linux/macOS/Windows/Android/iOS/WASM; xcframework and Android JNI samples in tree [Verified] | 60 CMake toolchains incl. Android, iOS, HarmonyOS, Jetson, QNX, ESP32, WASM [Verified] | Android/iOS/macOS/Linux/Windows-ARM64/OpenHarmony/WASM/RISC-V; CUDA sm87 present [Verified] | Android AAR, Apple SwiftPM xcframeworks, bare-metal Arm, Zephyr, ESP32, RISC-V, Linux, Windows, WASM [Verified] | + +Take-aways for edge: only the four runtimes reach Android GPU/NPU and Apple GPU/ANE. vLLM reaches ARM CPU and Jetson-class CUDA (untested in CI), and vLLM-Omni does not reach CPU at all. That last point is an **architectural** constraint only in the sense that vLLM-Omni's platform layer requires a device-backed worker; adding a CPU platform is a missing implementation, not a redesign (see §9). + +## 3. Kernel layer + +| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch | +|---|---|---|---|---|---|---| +| Operator set | torch ops + 132 CUDA custom ops + 73 CPU ops; attention backends by enum registry [Verified] | Inherits; adds diffusion attention registry (11 backends), fused/Triton ops for codecs with torch fallbacks [Verified] | 102 `ggml_op`s; per-backend `supports_op` vtables; `docs/ops.md` support matrix [Verified] | 110 layer types; 64 with Vulkan kernels [Verified] | 186 schema op types; CPU registers 75, Metal 36, OpenCL 39, Vulkan 35/26, QNN 41; geometry decomposition to Raster/Loop reduces what a backend must implement [Verified] | 209 portable + 23 optimized + 19 quantized kernels; LLM custom ops (`sdpa_with_kv_cache`, `update_cache`, ...) [Verified] | +| Dispatch | `CustomOp.dispatch_forward` per platform, `vllm.ir` provider priority, `MPLinearKernel.can_implement` chain [Verified] | Adds `forward_npu/musa`, per-role diffusion attention selection [Verified] | `ggml_backend_sched` assigns each node to the highest-priority backend whose `supports_op` is true [Verified] | `Layer_final{layer_cpu, layer_vulkan}` per layer, feature masks per layer [Verified] | Backend creator map; per-op CPU retry on failure [Verified] | Static kernel table resolved at `Method::init`; delegate calls are opaque blobs [Verified] | +| Quantization | AWQ/GPTQ/FP8/compressed-tensors/ModelOpt/MXFP4/torchao/...; ARM CPU: W8A8 (oneDNN) and W4A8 (KleidiAI) only [Verified] | Scoped to the AR LM stage; non-AR stages BF16 "not validated" (repo doc) [Verified] | 43 `ggml_type`s (Q4_0..IQ*, TQ, MXFP4); dequant fused into matmul; CPU repack + KleidiAI [Verified] | fp16/bf16, int8 PTQ, LLM block quant W4/W6/W8A8 (CPU only) [Verified] | Export-time W2/W3/W4/W8 block quant (default W4 block-64), runtime int8 dynamic activation; KV int8/TQ3/TQ4; GPU low-bit GEMV [Verified] | torchao 8da4w/4w/8w, SpinQuant, QAT+LoRA; PT2E quantizers per delegate; KV int8 [Verified] | +| Compilation / graph capture | `torch.compile` modes + CUDA graph modes, all at warm-up [Verified] | Per-stage compile config; talker-MTP graphs; code2wav stateful graphs per chunk length; diffusion regional compile [Verified] | No JIT; CUDA graph capture; Metal runtime shader compile; Vulkan build-time SPIR-V; OpenCL kernel cache [Verified] | Runtime GLSL → SPIR-V via glslang with pipeline cache [Verified] | No JIT/no CUDA graph; tuning cache files [Verified] | AOT only; optional CUDA graph per method [Verified] | +| Fallback | Native torch path per `CustomOp`; Triton C++ fallbacks on CPU [Verified] | Torch fallbacks for Triton codec kernels [Verified] | Unsupported node → CPU with inserted copies [Verified] | Layer without Vulkan impl → CPU with download/upload [Verified] | Per-op CPU retry; NPU subgraph compile is all-or-nothing [Verified] | Non-delegated ops stay portable CPU kernels; missing kernel → load error [Verified] | +| Extension | Platform plugin, `CustomOp.register_oot`, `vllm.ir.register_impl`, `register_quantization_config` [Verified] | `register_diffusion_backend`, TTS adapter registry, model/pipeline registries [Verified] | New backend = vtables + `GGML_BACKEND_DL_IMPL`; custom ops `ggml_custom_4d` [Verified] | `Net::register_custom_layer`; custom Vulkan shader op [Verified] | `MNNInsertExtraRuntimeCreator`; plugin ops [Verified] | New delegate = `BackendInterface` + partitioner; custom ops via YAML + kernel registration [Verified] | + +## 4. Model runner + +| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch | +|---|---|---|---|---|---|---| +| Model format / conversion | HF safetensors directly; no conversion [Verified] | Same; plus per-family Python module code; no export tooling [Verified] | GGUF via `convert_hf_to_gguf.py` (+ separate mmproj GGUF) then `llama-quantize` [Verified] | `.param/.bin` via pnnx (PyTorch/ONNX) or legacy converters; LLM decoder needs seq-len-1 export + param patch [Verified] | `.mnn` FlatBuffer via MNNConvert (ONNX/TF/TorchScript) or `llmexport.py` (HF → MNN with fused attention/RoPE ops); GGUF import [Verified] | `.pte` via `torch.export` → `to_edge_transform_and_lower` → `to_executorch`; LLM CLI `export_llm`; HF via optimum-executorch [Verified] | +| Lifecycle | init device → load → profile → KV alloc → compile/warm-up → busy loop [Verified] | Same per stage, sequentially; Orchestrator readiness gate (default 300 s, see experiments) [Verified, measured] | load → init context (reserve) → decode; no warm-up API [Verified] | load_param/model → extractor; no warm-up; first extract compiles Vulkan pipelines [Verified] | `Llm::load` builds tokenizer, modules, clones decode module; optional tuning [Verified] | load program → `Method::init` (kernel resolution, delegate init) → execute [Verified] | +| Dynamic shapes | Native; CUDA graphs bucketed by batch size [Verified] | Same; code2wav shapes derived from chunk schedule [Verified] | Native; graph reuse when shapes match [Verified] | Native; shape expressions [Verified] | Bucketed module clones (prefill/decode); `shapeMutable=false` on GPU/NPU + chunked prefill [Verified] | Upper-bound dims at export; XNNPACK reshapes per call; QNN/Ethos-U static [Verified] | +| KV / state | Block-paged, prefix caching, hybrid managers, offload connectors, streaming-input sessions [Verified] | Inherits; adds hidden-state prefix cache, inter-stage KV transfer, per-session paged KV in `ARDiffusionEngine` [Verified] | Ring-buffer cells, seq bitsets, no paging, context shift, quantized KV, state save/load [Verified] | Contiguous per-head `Mat`, ×1.5 growth, batch-1 only, host-provided mask/RoPE [Verified] | Contiguous per-head, grows by 64, int8/TQ quant, mmap-to-disk, no paging [Verified] | In-graph mutable buffer `[B,H,max_ctx,D]`, runner tracks position; static-attention I/O mode for NPUs; experimental paged-like cache lib [Verified] | +| Multimodal stages | Encoders in worker under an encoder budget, embeddings scattered [Verified] | Encoders, talkers, codec/vocoder, diffusion each as stages or model-owned sub-modules [Verified] | `mtmd` encode chunk → `llama_decode(embd)`; audio-gen step API [Verified] | Separate `Net`s wired by the app (Piper: 5 nets) [Verified] | `Omni` class: vision/audio encoders, talker, DiT, vocoder threads [Verified] | `MultimodalRunner` method conventions (`vision_encoder`, `text_decoder`, ...); Voxtral TTS runner [Verified] | + +## 5. Scheduler + +Request/token/pipeline scheduling is separated from graph/operator/thread scheduling, as the task requires. + +| Level | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch | +|---|---|---|---|---|---|---| +| Request / token | Token-budget continuous batching, chunked prefill, preemption by recompute, async scheduling, priority, admission backpressure, abort [Verified] | Per-stage vLLM schedulers (`OmniARScheduler`, async variant) + `OmniGenerationScheduler`; **pipeline-level**: Orchestrator routes, pre-warms downstream requests parked in `WAITING_FOR_CHUNK`, propagates abort [Verified] | `libllama`: none (caller batches `seq_id`s). `llama-server`: slots, one worker queue, continuous batching, cancel, speculative [Verified] | N/A [Verified] | N/A; batch 1 baked into export; cancel only through wav callback or per-token stepping [Verified] | N/A in core; experimental `DecodeFirstScheduler` without executor; `llm_server` serializes; cooperative `stop()` [Verified] | +| Pipeline (multi-stage) | N/A (single model) | Yes: stage DAG, connectors, async chunk hand-off (Talker → Code2Wav in 25-frame chunks after a 1-frame first chunk) [Verified, measured] | App-level only (`mtmd` helper loops) [Verified] | App-level (Piper example) [Verified] | Built-in for Omni: thinker/talker interleave and codec-LM → DiT → vocoder worker threads with `WavChunk` queues [Verified] | App-level (Voxtral TTS runner chunk loop) [Verified] | +| Graph / operator / thread | CUDA graph dispatch, DBO micro-batching; OpenMP binding on CPU; no op scheduler [Verified] | Inherits | `ggml_backend_sched` 5-pass split, `ggml_threadpool` with cpumask/polling, separate batch/decode thread counts [Verified] | Lazy DAG pull, OpenMP intra-op, big/little affinity [Verified] | Linear command walk, own thread pool (max 2 tasks), big/little pinning [Verified] | Sequential instruction chain; pthreadpool sized by performant cores; delegate-internal threading [Verified] | +| Concurrency | Many requests per step | Many per stage, stage-parallel | Server slots; library caller-managed | One Extractor per thread | One `Llm` per thread | One `Method` at a time | +| Cancellation | Immediate at next step | Propagated across stages | Server cancel task; library abort callback (CPU/Metal) | None | Weak (callback return / stepping) | Token-boundary stop | + +## 6. I/O layer + +| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch | +|---|---|---|---|---|---|---| +| Tokenizer | HF `tokenizers`, Mistral, incremental detok [Verified] | Inherits; per-model speech tokenizers (12/25 Hz Qwen3-TTS, s3tokenizer, DAC, XCodec) [Verified] | In-tree SPM/BPE/WPM/UGM/RWKV [Verified] | None [Verified] | In-house SentencePiece/tiktoken/BERT/HF-BPE + vendored Jinja [Verified] | Submodule (uninitialized here): HF json, Tiktoken, SentencePiece, Llama2c, Tekken [Verified/Unknown internals] | +| Image / audio preprocessing | HF processors in API process; PyAV/soxr audio [Verified] | Adds Silero VAD, forced aligner, resamplers, soundfile encoders [Verified] | stb_image + slicing; miniaudio + mel/FFT [Verified] | `Mat::from_pixels`, Spectrogram layers; no audio I/O [Verified] | ImageProcess, per-family vision preprocess, fbank → `audio.mnn` [Verified] | `ImageProcessor`, exportable mel spectrogram, ASR runners [Verified] | +| Streaming | Token SSE, WS realtime audio input [Verified] | Token + audio chunk streaming, `/v1/audio/speech/stream`, duplex sessions [Verified] | Server SSE; audio-gen step API but CLI writes one WAV [Verified] | None [Verified] | `setWavformCallback` chunked audio (Qwen2.5-Omni); Qwen3-TTS single callback [Verified] | Token callback; Voxtral TTS `AudioChunkCallback` [Verified] | +| Application boundary | Python, OpenAI/Anthropic HTTP, gRPC, CLI; no C API/mobile [Verified] | Python `Omni/AsyncOmni`, HTTP + WS incl. OpenPI robot endpoint; no C API/mobile [Verified] | C API, CLI, server, Android JNI, Swift; no Python runtime [Verified] | C++/C API, Python, Android/iOS packaging [Verified] | C++, Python (`MNN.llm`), Android/iOS/HarmonyOS apps, HTTP wrappers [Verified] | C++ `Module`, Python, Android AAR, Swift packages, WASM [Verified] | + +## 7. Deployment complexity, conversion, extensibility + +| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch | +|---|---|---|---|---|---|---| +| Minimum deployable unit | Python env with torch 2.13 + CUDA/CPU wheel (≤500 MB wheel cap) [Verified] | vLLM env + vLLM-Omni + transformers/diffusers/soundfile/onnxruntime + per-family extras; 2 engine processes for a 2-stage TTS [Verified] | Single static binary; libs of ~0.9 MB (`ggml-base`) + 1.5 MB (`ggml-cpu`) + `libllama` measured here on x86 [measured] | `libncnn` (size deltas only in CI) [Unknown absolute] | Android core `.so` ≈ 800 KB, iOS static ≈ 12 MB repo-reported [repo-reported] | Core ≈ 50 kB (CI thresholds 45–52 kB) + delegates + kernels [repo-reported] | +| Cold start | Seconds to minutes (compile + graph capture) | **157–188 s observed** for Qwen3-TTS 2-stage on L20X with default CUDA graphs and `torch.compile` (sequential stage init; `enforce_eager`, per-stage `compilation_config`, and parallel init exist but were not measured) [measured, not a lower bound] | Sub-second to seconds (mmap) [Inferred from design; not measured here except mtmd run ≈ 8 s incl. 1 GB load] | Sub-second (first Vulkan pipeline compile aside) [Inferred] | Seconds (module clones) [Inferred] | Sub-second (static planning) [Inferred] | +| Conversion effort for a new HF model | None (Python model class) | Python model class + pipeline config + deploy YAML | Converter class in `conversion/` + graph class in `src/models/` + optional mmproj | pnnx export + possibly custom layers + param patching | `llmexport.py` mapper entry or ONNX path; fused ops emitted by exporter | `torch.export` of each sub-module with static/upper-bound shapes; partitioner per target; quantizer recipe | +| Adding a backend | Platform plugin (out-of-tree possible) | Same + omni platform overlay | ggml backend vtables (in-tree or dynamic) | Third slot in `Layer_final` (in-tree) | `MNNInsertExtraRuntimeCreator` (in-tree) | `BackendInterface` + partitioner (in- or out-of-tree) | +| Adding an op | `CustomOp` / `vllm.ir` | Same | `ggml_custom_4d` or new op enum across backends | `register_custom_layer` | Plugin op | YAML + kernel or delegate-internal | + +## 8. What vLLM-Omni inherits from vLLM and what it adds + +Inherited (used as-is or subclassed) [Verified in engines/vllm_omni.md]: + +- Engine processes and IPC: `EngineCoreProc` (subclassed as `StageEngineCoreProc`), `AsyncMPClient` over ZMQ, msgspec serialization. +- Scheduler: `Scheduler`/`AsyncScheduler` subclassed into `OmniARScheduler`/`OmniARAsyncScheduler`; token budgets, chunked prefill, preemption, prefix caching. +- KV cache manager and block pool; multimodal processor and caches; tokenizers; quantization configs; `torch.compile`/CUDA-graph machinery; platform plugin mechanism; OpenAI server scaffolding. +- Model layers for talkers (`Qwen2Model`/`Qwen3Model`, `ParallelLMHead`, paged attention). + +Added [Verified]: + +- Stage abstraction: `PipelineConfig` (LLM_AR / LLM_GENERATION / DIFFUSION) + `DeployConfig` YAML; `AsyncOmniEngine`, `Orchestrator`, `StageRuntime/StagePool`, headless remote replicas. +- `LLM_GENERATION` runner (`GPUGenerationModelRunner`: no logits/sampling, tensor outputs) and `OmniGenerationScheduler`. +- Diffusion runtime: `DiffusionEngine` with request/step schedulers, workers, 11-backend attention registry, offload, parallelism (TP/SP/CFG/HSDP/VAE). +- Connectors (`OmniConnector`: shared memory, Mooncake, Mori, Yuanrong) and the async-chunk protocol (`WAITING_FOR_CHUNK`, `OmniChunkTransferAdapter`). +- Model families: 15 TTS, 12 speech-to-speech/omni, 5 image, 4 video/world, 2 VLA pipelines; TTS adapter registry; audio encoders/resamplers; OpenPI robot WebSocket; duplex sessions. +- Monkeypatches of vLLM internals (`patch.py`) and injection of omni model classes into vLLM workers via the `vllm.general_plugins` entry point. + +Consequence: vLLM-Omni is not separable from vLLM's process model. It imports 284 `vllm.*` modules and patches request/output classes; "vLLM-Omni without vLLM" is not a meaningful target. What *is* separable is the **model code of individual stages** (see [engines/vllm_omni_models.md](engines/vllm_omni_models.md) §5: decoders, codecs, action heads are plain torch) and the **pipeline semantics** (stage graph, chunk sizes, abort rules), which is what the edge design re-implements. + +## 9. Architectural constraints versus missing implementations + +| Engine | Architectural constraint (would need a redesign) | Missing implementation (could be added without changing the architecture) | +|---|---|---| +| vLLM | Multi-process engine with ZMQ IPC and a Python control loop; server-class dependency set (torch, transformers) | Jetson wheels/CI; Android/Vulkan/QNN (would be new platform plugins, but the torch dependency remains) | +| vLLM-Omni | Every AR/generation stage is a vLLM engine process and diffusion stages are a worker process or inline; the default inter-stage transport is host-staged shared memory (the Mooncake/Mori transfer-engine connectors have device-memory fast paths, and Mori's `xgmi` backend covers intra-node AMD GPU-to-GPU per `deploy/qwen3_omni_moe_mori_intranode.yaml`; none is validated for an edge target); start-up = per-stage load + optional compile/capture | CPU platform; a device-direct connector for single-GPU/edge co-location (repo docs list D2D as roadmap); action keys in `OmniPayload`; CPU/edge deploy YAML overlays; a codec-token streaming wire contract (the speech stream endpoint consumes PCM today) | +| llama.cpp | Single-graph-per-arch C++ model definitions (each new architecture is C++ work); no request scheduler in the library | Server TTS endpoint; batched audio generation; abort callback on GPU backends; VLA action head (proposal in report) | +| ncnn | No LLM runtime, no tokenizer, batch-1 KV cache, host-provided masks/RoPE; app owns all pipeline logic | NPU delegates; more Vulkan ops (Embed, MatMul, GLU, RNN); LLM export tooling in-tree | +| MNN | Batch 1 baked at export; static shapes on GPU/NPU; one `Llm` per conversation; no request queue | Streaming callback for Qwen3-TTS (single callback today); Python TTS API; more TTS families; VLA (nothing yet) | +| ExecuTorch | Everything is AOT: no runtime op fallback across devices, static kernel table, upper-bound shapes, one `Method` at a time | In-tree batching executor; mobile TTS example; VLA models; generic TTS runner | + +## 10. TTS and VLA capability matrix + +| Capability | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch | +|---|---|---|---|---|---|---| +| TTS model families in tree | none | 15 (Qwen3-TTS, CosyVoice3, Fish S2 Pro, MOSS, IndexTTS-2, VoxCPM2, GLM-TTS, Higgs, OmniVoice, Ming, Voxtral TTS, Gepard, dots, Audex, ...) | Qwen3-TTS, Pocket-TTS (`MTMD_GEN_AUDIO_TYPE_*`), legacy WavTokenizer arch | Piper/VITS example | Qwen3-TTS (`Llm::generateTTS`), Qwen2.5-Omni talker, `mnn_tts` SDK (Bert-VITS2, Supertonic, Piper), sherpa-mnn (Kokoro, Matcha, VITS) | Voxtral-4B-TTS example (streaming), Supertonic (MLX only) | +| Streaming audio out | N/A | Yes, 1-frame first chunk then 25-frame chunks; measured TTFA ≈ 28 ms on L20X | Step API (app-level streaming possible), CLI writes one WAV | App-level | Qwen2.5-Omni chunked callback; Qwen3-TTS single callback | Voxtral chunk callback (25 frames, initial 5) | +| Codec/vocoder state across chunks | N/A | Explicit (conv ctx, 72-frame window, ref-prefix cache) for Qwen3-TTS; MOSS ring KV; CosyVoice3 re-decodes cumulative history | Qwen3-TTS code2wav 72-frame windows | App-owned | App/engine-owned worker queues | Runner-owned (left context 25) | +| VLA models in tree | OpenVLA (image-only, discrete action bins) | GR00T N1.7, π0, DreamZero, InternVLA-A1 (offline), Cosmos3 action modes; OpenPI WebSocket serving | none (vision encoders + `embd` input only) | none | none (vision exporters + DiT runtime) | none (VLM runners, DiT on QNN) | +| Structured (proprioceptive) inputs | none (`EmbedsPrompt` only) | Via diffusion sampling-param `extra_args["robot_obs"]`, not through `OmniPayload` | `llama_batch.embd` | Any `Mat` input | Any `VARP` input | Any tensor input | +| Action head type | discrete tokens | Flow/diffusion heads (4–16 Euler/UniPC steps), stateless per step, batch 1, eager | Would be a sidecar graph (proposal) | Host denoise loop per extract | Host denoise loop; DiT runtime exists | Loop inside `.pte` (`while_loop` supported) or runner loop | + +## 11. Measured data points that anchor the comparison + +From [experiments.md](experiments.md), this host only (x86 Xeon 8480C; L20X): + +| Measurement | Value | What it tells the edge plan | +|---|---|---| +| vLLM-Omni Qwen3-TTS 1.7B, 1 GPU, batch 1: TTFA p50 / p95 | 27.7 / 30.7 ms | First-audio is nearly free once graphs are warm, but the default 1→25 schedule leaves an ≈80 ms playback gap (80 ms of audio at 30 ms, next chunk at 190 ms); the deploy YAML's `codec_chunk_ramp`/`codec_chunk_adaptive` options exist to remove that cliff. An edge port must keep the chunk semantics configurable and gate on uninterrupted playback, not on first-chunk arrival. | +| Same: steady cadence per 25-frame (2 s) chunk | ≈160 ms | About 6.4 ms per codec frame for talker + code predictor + vocoder; the edge budget is 80 ms per frame. | +| Same: RTF | 0.08–0.10 | ~10× real-time headroom on a data-center GPU. No device extrapolation is made here: predictor and decoder scaling on edge hardware is unmeasured. | +| Same: engine initialization (default config) | 157–188 s | Observed with CUDA graphs and `torch.compile`, cold and warm caches; the minimum with `enforce_eager`/parallel init is unmeasured. Either way the edge design precompiles/exports ahead of time. | +| llama.cpp CPU, 409 M-param LLM, Q4_0, 4 threads: decode | 135.6 tok/s | Backbone-only microbenchmark. A Qwen3-TTS frame also needs 15 code-predictor sub-steps (`qwen3_code_predictor.py`) and the decoder, none of which `llama-bench` measures; it bounds only the talker backbone's share of the 80 ms frame budget. | +| llama.cpp CPU, SigLIP-class vision encoder, one 512 px tile, 8 threads | ≈0.6–1.2 s | Vision encoding, not the LLM, dominates per-frame VLA cost on CPU; tile count must be pinned to 1 per camera. | + +## 12. Recommendation summary (detailed in the design documents) + +- Treat vLLM-Omni as the **reference implementation and cloud/edge-server tier**, not as code to ship on a phone. What transfers to edge is its stage graph, chunk schedule, state ownership rules, abort semantics, and API shapes. +- For **TTS on edge**, llama.cpp and MNN already run the same Qwen3-TTS family; ExecuTorch has a streaming TTS runner pattern (Voxtral). Choose per target: llama.cpp (ARM CPU, Apple Metal, Adreno OpenCL, Hexagon), MNN (Android CPU/OpenCL/QNN, iOS Metal/CoreML), ExecuTorch (CoreML/MLX, QNN, XNNPACK with the strongest AOT story), ncnn (only as a vocoder/VITS-class runtime). +- For **VLA on edge**, no runtime has a policy model in tree; the closest building blocks are ExecuTorch (static memory, zero-copy I/O, `while_loop` for the denoise loop, QNN/CoreML delegates) and MNN (DiT runtime, vision exporters, NPU delegation). llama.cpp is viable for the VLM backbone but not for the action head today; ncnn fits only the vision encoder or a small MLP head. +- See [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) for the strategy comparison, [tts_edge_design.md](tts_edge_design.md) and [vla_edge_design.md](vla_edge_design.md) for the designs, and [implementation_roadmap.md](implementation_roadmap.md) for phases and acceptance criteria. diff --git a/docs/engines/edge_device_simulation.md b/docs/engines/edge_device_simulation.md new file mode 100644 index 000000000..71b75daa7 --- /dev/null +++ b/docs/engines/edge_device_simulation.md @@ -0,0 +1,97 @@ +# Simulating edge devices on this host, and why the CPU path is slow + +Companion to [edge_engine_comparison.md](edge_engine_comparison.md). Measured 2026-09-09; raw data under [experiments/edge_engine_compare/results/gpu_emu](experiments/edge_engine_compare/results/gpu_emu/) and [results/cpu_attrib](experiments/edge_engine_compare/results/cpu_attrib/), driver scripts `gpu_emu_chain2.sh` (MPS-capped runs) and the branch's `hw_emulation.py`. + +## 1. What this GPU is, and what can be emulated without root + +`nvidia-smi` reports the eight cards as "NVIDIA L20X, Ada Lovelace, 143,771 MiB", but the measured dense bf16 matmul throughput is **804 TFLOPS** (8192³, 20 iterations) with 132 SMs at 1980 MHz and 140 GiB, which is H200-class silicon, not an Ada L20 (≈120 TFLOPS). The comparison and the branch's results should therefore be read as **H200-class** numbers; the label in earlier documents is the driver's. + +| Edge property | Emulable here? | How | Not covered | +|---|---|---|---| +| Fewer SMs (Orin AGX 16 SMs, Orin NX 8, Thor 20 vs 132) | **yes** | user-level MPS daemon + `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` (verified: 12 % cap → 100.6 TFLOPS, ratio 0.125). Two traps: the Unix-socket path of `CUDA_MPS_PIPE_DIRECTORY` must be < 108 bytes, and clients inside the MPS namespace must use `CUDA_VISIBLE_DEVICES=0` even when `gpu run` assigned another index | lower SM clocks (Orin 1.3 GHz vs 1.98 GHz) | +| Small VRAM / unified memory budgets | yes | branch `deploy_profile="auto"` with `VLLM_OMNI_HW_PROFILE` overrides: absolute KV budgets, eager mode, small batches (`hw_emulation.py` jetson_orin_8g / 32g / cuda_discrete_8g) | unified-memory bandwidth sharing with the CPU | +| Weak host CPU (Orin: 8–12 Cortex-A78AE at 2.2 GHz) | partly | `taskset` to 4–6 Xeon cores for the whole engine; x86 cores are still ≈2× faster per core | ARM ISA, big.LITTLE | +| Memory bandwidth (HBM 4.8 TB/s vs LPDDR5 102–273 GB/s) | **no** | analytic projection only (§4) | – | +| Clock / power caps, MIG partitions | no | `nvidia-smi -lgc/-pl` and MIG need root; MIG is disabled on these cards | – | +| Thermal throttling, CPU/GPU memory contention | no | – | – | + +## 2. GPU: sensitivity to SM count (MPS caps, H200, 0.6B CustomVoice, edge profile) + +Same 12 prompts, one utterance at a time, `deploy_profile="edge"`, 0.6B CustomVoice; MPS `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` = share of the 132 SMs available to every process of the deployment (both stages). llama.cpp rows: `llama-tts` CUDA build, all layers offloaded, same cap. The "Jetson-like" row combines a 12 % SM cap (16 SMs ≈ Orin AGX), 6 host cores for the whole engine, and the branch's `jetson_orin_8g` hardware profile (eager mode, absolute 8 GB budget). + +| SM share (≈ SMs) | vLLM-Omni talker step ms/frame | vLLM-Omni TTFA p50 / p90 | vLLM-Omni RTF | llama.cpp talker ms/frame | llama.cpp vocoder ms/frame | llama.cpp RTF | +|---|---|---|---|---|---|---| +| 100 % (132) | 6.0 | 41–46 / 49 ms | 0.10 | 17.4 | 3.8 | 0.28 | +| 25 % (33) | 6.5–7.0 | 43 / 49–53 ms | 0.08–0.09 | 23.1 | 2.1 | 0.33 | +| 12 % (16, Orin AGX class) | 9.1 | 52 / 58 ms | 0.12 | 33.7 | 4.7 | 0.50 | +| 6 % (8, Orin NX class) | 9.3 | 58 / 66 ms | 0.13 | 48.0 | 4.1 | 0.68 | +| Jetson-like combined: 12 % + 6 host cores + 8 GB profile (eager) | ≈13 (from RTF) | 160 / – ms | 0.50 | – | – | – | +| (reference) 8 GB profile alone, uncapped, from §8.5 | – | 181 ms | 0.57 | – | – | – | + +Reading: with CUDA graphs and one utterance in flight, the talker step is latency-bound: cutting the SMs to 6 % of an H200 costs vLLM-Omni only +55 % per frame (6.0 → 9.3 ms) and +15 ms of TTFA, and RTF stays ≈0.13. llama.cpp's kernels are launch-bound per layer and lose 2.8× at the same cap (17 → 48 ms per frame). The combined Jetson-like case shows where the real cost sits: forcing eager mode (the 8 GB profile disables CUDA graphs so the small budget is not consumed by graph pools) raises the frame time to ≈13 ms and TTFA to 160 ms regardless of the SM cap, i.e. **on a small-memory Jetson the branch should keep graph capture for the talker/predictor and spend the memory elsewhere**, which is the next tuning item for `deploy/edge/hardware/jetson.yaml`. What SM caps cannot show is the memory-bandwidth term, projected in §4. + +## 3. Where a GPU frame goes (synchronized attribution, uncapped) + +`VLLM_OMNI_STEP_STATS_SYNC=1` synchronizes the device around the model-level timers (adds ≈3 ms per step of sync overhead, so `core.step` reads 9.6 ms here instead of the 6.0 ms measured without sync): + +| per talker step (= one 80 ms frame) | ms | +|---|---| +| model forward (28-layer talker, CUDA graph) incl. sampling hook | 1.8 | +| sampler | 0.6 | +| talker-MTP graph (15 code-predictor sub-steps + embeddings, one CUDA graph) | inside the forward's 1.8 ms (graph replay; the separate `model.mtp_graph_ms` timer measures 0.04–0.43 ms of launch time without sync) | +| scheduler (`schedule` + `update_from_output`) | 0.2 | +| remainder: input preparation, D2H of codes/hidden, Python bookkeeping | ≈3–4 | + +The GPU compute for a frame is a couple of milliseconds; more than half of the 6 ms step is host-side runner work, which is what a slow ARM host CPU would stretch first. + +## 4. Analytic projection to Jetson-class devices + +The talker step is memory-bound at batch 1: every frame streams the talker weights once (0.6B bf16: 1.2 GB) and the code-predictor weights fifteen times (5 layers × 1024 hidden ≈ 160 MB each pass, 2.4 GB per frame). Using the measured H200 split and datasheet bandwidths (weights from the HF config; bandwidths: H200 4.8 TB/s, Orin AGX 204.8 GB/s, Orin NX 102.4 GB/s, Thor 273 GB/s): + +| device | weight streaming per frame (talker + 15 predictor passes) | host-side runner work (≈4 ms on Xeon, scaled by single-core speed) | projected ms/frame | projected RTF (80 ms frame) | +|---|---|---|---|---| +| H200 (measured) | 0.75 ms | ≈4 ms | 6.0 (measured) | 0.10 (measured) | +| Jetson AGX Orin 64 GB | 17.6 ms | ≈8 ms | ≈26 | ≈0.33 | +| Jetson Orin NX 16 GB | 35 ms | ≈8 ms | ≈43 | ≈0.55 | +| Jetson Thor | 13 ms | ≈6 ms | ≈19 | ≈0.24 | + +These are floors from bandwidth plus measured host overhead, not measurements: kernel launch latency at small SM counts, unified-memory contention with the code2wav stage on the same memory bus, and thermal limits push them up. The MPS 12 % run (§2) is the compute-side check of the Orin AGX case (16 of 132 SMs); a 4-bit talker (llama.cpp Q4_0 runs at 3.1 ms/token on 16 Xeon threads for the same backbone) would cut the streaming term by ≈3× on all rows. **Conclusion:** the branch's talker stays real-time on Orin AGX / Thor with 2–4× headroom by this model, is marginal on Orin NX, and its 15-pass code predictor is the term to attack (see §5). + +## 5. Why the CPU path is slow + +### 5.1 Measured attribution (vLLM-Omni CPU platform, 0.6B CustomVoice, 12+4 threads on one socket, bf16 with AMX) + +| per frame (one talker step) | ms | share of the 80 ms frame | +|---|---|---| +| **code predictor loop** (15 sub-steps × 6.4 ms, 5-layer 1024-hidden model, batch 1) | **97.1** | 121 % | +| talker forward (28 layers, 0.45 B params) | 25.4 | 32 % | +| sampler (top-k/top-p on a 3072-way vocabulary, penalties, CPU) | 9.9 | 12 % | +| scheduler + bookkeeping | ≈2 | 2 % | +| **engine-core step** | **133.9** | **167 %** | + +(`core.step` 134 ms here vs 147 ms in the earlier sweep: the attribution run had the host to itself.) The code2wav stage runs concurrently in its own process and does not add to this cadence. + +### 5.2 The same split in llama.cpp (native C++, same host, 16 threads) + +`llama-bench` on the converted talker GGUF gives the backbone alone at **126.7 tok/s f16 (7.9 ms/token)** and **325.8 tok/s Q4_0 (3.1 ms/token)**, yet `llama-tts` needs **85–97 ms per frame**: ≈80 ms of every frame is the fifteen code-predictor passes (≈5.3 ms each) plus sampling. llama.cpp's predictor pass is only ≈20 % cheaper than vLLM-Omni's 6.4 ms even though it is hand-written C++ with no Python — so the cost is not "Python overhead"; it is the structure of the work. + +### 5.3 Root causes, in order + +1. **Sixteen sequential forward passes per 80 ms frame.** One talker pass plus fifteen residual-codebook passes, each depending on the previous sample. Real time needs the whole chain under 80 ms, i.e. **< 5 ms per pass including sampling**. Both engines sit at 5.3–6.4 ms per predictor pass on 16 Xeon threads. +2. **Each pass is latency-bound, not bandwidth-bound.** A predictor pass reads ≈160 MB of bf16 weights, which is ≈1.6 ms at the ≈100 GB/s one socket-half delivers; the measured 6.4 ms is 4× that. The rest is per-operator cost: ≈100 small kernels per pass (5 layers × QKV/attention/RoPE/norms/MLP/gate) each paying OpenMP fork-join across 12 threads, cache-cold tiny GEMVs, and (in PyTorch) operator dispatch. The talker's 28-layer pass is at 2× its bandwidth floor (25 ms vs 12 ms) for the same reason. +3. **Sampling costs 10 ms on CPU.** vLLM's sampler applies penalties, top-k and top-p over the padded vocabulary per step with several full-vocabulary passes; on GPU this is 0.6 ms. The fifteen residual codebooks are also sampled (Gumbel top-k) inside the predictor loop. +4. **Threads stop helping at 16.** 16 → 32 threads changed the frame time by 8 % (147 → 136 ms) because the work is latency-bound (item 2) and OpenMP synchronization grows with thread count; the two stages also compete for the same memory channels. +5. **bf16/AMX is the only CPU precision on the branch.** There is no int8/int4 talker path in vLLM's CPU backend for this model; llama.cpp's Q4_0 backbone shows the bandwidth term can shrink 2.5×, but that only helps the part that is bandwidth-bound (talker pass, ≈25 → ≈12 ms), not the predictor's overhead floor. +6. **Vocoder**: not on the vLLM-Omni critical path (concurrent stage), but in llama.cpp the BigVGAN decoder costs a further 55 ms per frame serially, which is why its end-to-end RTF matches vLLM-Omni's despite the faster talker. + +### 5.4 What would make CPU real-time, with expected effect + +| change | where | expected per-frame effect (0.6B, 16 threads) | evidence | +|---|---|---|---| +| Fuse each predictor pass into one kernel / one OpenMP region (C++ or a compiled graph), drop per-op dispatch | predictor | 6.4 → ≈2–3 ms per pass, i.e. −50…−65 ms per frame | bandwidth floor 1.6 ms; llama.cpp's eager C++ is already at 5.3 ms, so fusion (not language) is the lever | +| Greedy or cheap top-k for the 15 residual codebooks; skip penalties for codec ids | sampler + predictor | −8 ms (sampler) and part of each pass | GPU sampler is 0.6 ms for the same work | +| Int8/int4 talker weights (AMX int8 or a GGUF-style kernel) | talker pass | 25 → ≈10–12 ms | llama-bench Q4_0 3.1 ms vs f16 7.9 ms per token | +| `torch.compile` (inductor) on the CPU runner | whole step | −6 % measured (`cpu16_inductor` in §8.4) | measured | +| Architectural: fewer residual codebooks per pass (predict 2–4 codebooks per pass) or a parallel/non-autoregressive predictor | model | 15 → 4–8 passes per frame | requires retraining or a different checkpoint; out of scope for the branch | + +Adding the first three rows gives ≈134 → ≈45–60 ms per frame, i.e. real time with little margin on 16 Xeon threads; on a 4–8-core ARM box (≈2–3× slower per core, see the emulated `x86_cpu_avx2` case at RTF 12) the architectural row is the only one that reaches real time. That matches the study's original conclusion: CPU-only real-time TTS with this model family needs a lighter talker/predictor, not a faster runtime. diff --git a/docs/engines/edge_engine_comparison.md b/docs/engines/edge_engine_comparison.md new file mode 100644 index 000000000..a7d53cd61 --- /dev/null +++ b/docs/engines/edge_engine_comparison.md @@ -0,0 +1,121 @@ +# Edge engine comparison: Qwen3-TTS on vLLM-Omni (edge branch) vs native edge runtimes + +Measured on 2026-09-09 on the same host as the rest of this study (Xeon 8480C, one NVIDIA L20X through `gpu run --gpus 1`), same 12 English prompts (`vllm-omni/benchmarks/tts/prompts_12.txt`), same model family (Qwen3-TTS-12Hz, 0.6B and 1.7B), same reference speaker wav, one utterance at a time. Raw results, driver scripts and logs live in [experiments/edge_engine_compare/](experiments/edge_engine_compare/); the merged table is [results_summary.md](experiments/edge_engine_compare/results_summary.md) (regenerate with `python summarize_compare.py --out results_summary.md`). + +## 1. What was compared, and what could not be + +| Engine (checkout) | Qwen3-TTS path | Status here | +|---|---|---| +| vLLM-Omni `edge/omni-edge` (this branch) | two-stage talker → code2wav pipeline, streaming chunks, edge profile; CPU platform from WP3 | measured on GPU and CPU | +| llama.cpp `e71b80510` | `llama-tts` + `libmtmd` (talker GGUF + mmproj holding the speaker encoder and the codec decoder; `conversion/qwen3tts.py`) | built out of tree (CPU and CUDA), models converted locally from the cached HF checkpoints (f16, Q8_0, Q4_0), measured on GPU and CPU | +| MNN `bef71b97` | `transformers/llm/engine/demo/qwen3_tts_demo` + `llmexport.py` | built (`MNNConvert`, `qwen3_tts_demo`), but the exporter imports the external `qwen_tts` package (the QwenLM/Qwen3-TTS GitHub repo), which is not on this host; cloning it is a download and was not done without permission. **Not measured.** | +| ExecuTorch `cfc6ecd516` | no Qwen3-TTS runner (the streaming TTS runner is Voxtral); the branch's decoder export targets it | needs `pip install executorch` (permission gate). **Not measured.** | +| ncnn `3b7bdba7` | no TTS model | not applicable | + +Model/task parity: llama.cpp and MNN only run the **Base** checkpoints with a reference speaker (x-vector cloning); the branch's headline numbers are for **CustomVoice** (preset speakers). So this comparison runs vLLM-Omni on the *same* Base checkpoints with the *same* reference wav in `x_vector_only_mode`, and quotes the CustomVoice numbers separately where the reference-audio cost matters. The reference wav is a 4.5 s utterance synthesized earlier by the two-stage server; it drives timing only, not quality. + +## 2. Metric definitions + +- **TTFA**: vLLM-Omni: measured time from request submission to the first streamed PCM chunk (p50 over 12 prompts × repeats). llama.cpp: the tool is not streaming (all frames, then one vocoder pass, then the WAV), so the column is an **estimate**: prompt eval + 2 talker frames + 2 frames' share of the vocoder time (the edge profile's first chunk is 2 frames). llama.cpp's speaker encoding happens at load time and is excluded; vLLM-Omni's per-request x-vector extraction is included in its Base TTFA. +- **Talker ms/frame**: llama.cpp: generation time / frames from its own timing line. vLLM-Omni: the stage-0 engine-core step from the WP8 counters (one step = one frame, includes the 15-codebook predictor). +- **Vocoder ms/frame**: llama.cpp's vocoder time / frames. vLLM-Omni's code2wav runs concurrently in its own stage, so its cost does not add to the frame period; it is not tabulated. +- **RTF**: total processing wall time / seconds of audio (llama.cpp: its "total" line; vLLM-Omni: request wall time / audio). +- **Startup**: vLLM-Omni: engine init to ready (`init_s`, both stages). llama.cpp: process start + weight load + speaker encoding (`wall − total`, per invocation; the tool reloads everything for every utterance). +- **Memory**: peak RSS of the process tree (both), peak per-process GPU memory (vLLM-Omni, `nvidia-smi` samples; llama.cpp GPU memory was not sampled). + +## 2b. Mobile (real devices via Qualcomm AI Hub, 2026-09-10) + +Measured on Samsung Galaxy S24 (Snapdragon 8 Gen 3) and Galaxy S25 (Snapdragon 8 Elite): QNN context binaries on the Hexagon NPU, TFLite on the Adreno GPU and the phone CPU, AI Hub on-device medians. Details, job ids and fidelity in [mobile_htp_simulation.md §5](mobile_htp_simulation.md); export code in `vllm_omni/edge/qnn_export.py` and `decoder_export.py`. + +| component (0.6B, real weights), per 80 ms frame | Hexagon NPU | Adreno GPU | phone CPU | +|---|---|---|---| +| talker decode step (v2 export), fp16 | **15.8 ms** | 29.9 ms | 29.8 ms | +| code predictor, 15 cached steps, fp16 | **33.9 ms** | 65.0 ms | 72.3 ms | +| Code2Wav vocoder, fp16 | 182 ms | **21.7 ms** | 109 ms | +| Code2Wav vocoder, fp32 (numerically exact) | – | **45.8 ms** | 109 ms | + +Best placement: talker + predictor on the NPU, vocoder on the GPU. The two are separate stages exchanging codec tokens, so they pipeline: **49.7 ms per 80 ms frame, RTF 0.62** (96 ms, RTF 1.19 if run serially with the exact vocoder). Fidelity: NPU fp16 39.2 dB (vocoder) / 48.3 dB (predictor); GPU fp32 108.4 dB; GPU fp16 27.5 dB. Every automatic int8 build is 5–17× faster and numerically broken (vocoder −4.3 dB, predictor 3.1 dB), so int8 is not counted. + +Runtime comparison on the same graphs and phone (per frame): the talker step costs 15.8 ms on a QNN context binary, 16.3 ms on TensorFlow Lite and 16.6 ms on ONNX Runtime — all NPU paths converge because they all end in QNN, so the export layout matters and the runtime does not. They separate elsewhere: the vocoder is 21.7 ms on TFLite/GPU, 122.5 ms on ONNX Runtime/GPU, 109–124 ms on the phone CPU, and cannot run on the NPU under TFLite or ONNX Runtime at all (fails after compiling; out of device memory). + +Reading: with the heterogeneous split the branch's 0.6B pipeline is **real time on a 2024–2025 flagship phone, in faithful precision, without quantization**. llama.cpp and MNN have no Hexagon NPU path for Qwen3-TTS at all; llama.cpp's CPU-only pipeline extrapolates above RTF 2 on phone cores. The mistake to avoid is sending everything to the NPU: that is 3.7× slower than the split. + +## 2c. Mobile inference engines on the same graphs (x86 CPU, 2026-09-10) + +MNN, ncnn and ExecuTorch were the three engines this study could not reach earlier. All three are now installed +(`.venvs/edge-engines`) and fed the *same* exported graphs as the QNN/TFLite/ONNX-Runtime runs: the v2 talker decode +step (256-token cache), the code-predictor step (16-token cache) and the Code2Wav chunk (25 frames of audio from a +97-frame window). Median of 15 runs after 3 warm-ups, pinned to 8 or 16 physical cores of the Xeon 8480C, fp32. +Accuracy is signal-to-noise against the PyTorch output for the same input; the harness is +`experiments/edge_engine_compare/qnn/engines/bench_engines.py`. + +| graph | PyTorch 2.14 | ONNX Runtime 1.29 | MNN 3.x | ExecuTorch (XNNPACK) | ncnn | +|---|---|---|---|---|---| +| code predictor step, 8 / 16 threads | 4.47 / 3.30 ms | 4.27 / **2.51** ms | **3.69** / 4.13 ms | 6.89 / 5.01 ms | segfault | +| talker decode step, 8 / 16 threads | 35.96 / 20.32 ms | **26.40 / 17.04** ms | 37.46 / 31.39 ms | 73.04 / 61.87 ms | segfault | +| Code2Wav chunk (2 s audio), 8 / 16 threads | **957 / 590** ms | 1562 / 1248 ms | 1621 / 1059 ms | 1059 / 633 ms | 3538 / 2331 ms, silent | +| accuracy vs PyTorch (16 threads) | reference | 86–114 dB | 85–109 dB | 84–107 dB | NaN inside the graph | + +Reading: + +- **ncnn cannot run this model.** pnnx converts all three graphs at its default optimisation level, but the two + transformer steps segfault inside `Extractor::extract` and the vocoder fills with NaN partway through, which the + final `clamp(-1, 1)` turns into constant silence — the timing column above is therefore meaningless for it. At + `optlevel=0` the graph will not even load (`layer F.linear not exists or registered`). This is a converter-quality + problem in the ncnn toolchain for these graphs, not a missing runtime feature. +- **MNN and ExecuTorch both work and are numerically faithful** (85–109 dB, i.e. fp32 arithmetic differences only). + MNN is the fastest engine on the small predictor step at 8 threads; ExecuTorch is consistently the slowest on the + transformer steps (2–3× ONNX Runtime) but mid-pack on the vocoder. +- **ONNX Runtime is the best CPU engine for the transformer steps** (1.4–1.8× MNN and ExecuTorch); **PyTorch is the + best for the vocoder**, where its fused convolution/activation kernels beat every exported runtime. +- Caveat that matters: this is x86. MNN, ncnn and ExecuTorch are tuned primarily for ARM (NEON, dotprod, i8mm), so + this ranks the engines *on this host*, not on a phone. The phone-side ranking is in §2b, where the same graphs run + on the actual accelerators. +- MNN's Interpreter (C-style) API silently mis-feeds tensors of these ranks and produced −4 dB output; the numbers + above come from its Module/expr API. Anyone benchmarking MNN on multi-input graphs should check parity first. + +## 3. GPU (one L20X) + +| engine | model | TTFA | talker ms/frame | vocoder ms/frame | RTF | startup | peak RSS | +|---|---|---|---|---|---|---|---| +| llama.cpp CUDA, all layers | 0.6B Base f16 | 109 ms (est.) | 17.7 | 3.6 | 0.31 | 4.5 s per invocation | not sampled | +| llama.cpp CUDA, all layers | 0.6B Base Q4_0 | 125 ms (est.) | 16.8 | 2.4 | 0.28 | 3.4 s | not sampled | +| llama.cpp CUDA, all layers | 1.7B Base f16 | 146 ms (est.) | 17.9 | 2.1 | 0.28 | 6.0 s | not sampled | +| vLLM-Omni edge profile | 0.6B Base, x-vector | 561 ms p50 (incl. per-request x-vector: chunks then 27–43 ms apart) | 6.0 | – | 0.20 | 148 s | 9.0 GiB + 36.6 GB GPU | +| vLLM-Omni edge profile | 1.7B Base, x-vector | 374 ms p50 (same) | 6.4 | – | 0.20 | 149 s | 8.8 GiB + 37.0 GB GPU | +| vLLM-Omni edge profile | 0.6B CustomVoice (no ref audio) | 41–54 ms p50 | 6.0 | – | 0.10 | 87 s | 8.7 GiB + 37 GB GPU | +| vLLM-Omni edge profile | 1.7B CustomVoice (no ref audio) | 44–46 ms p50 | 6.4 | – | 0.10 | 88 s | 7.8 GiB + 37 GB GPU | + +Reading: +- **Steady-state generation** is where vLLM-Omni wins on the GPU: 6 ms per codec frame (CUDA graphs, talker-MTP graph, compiled code predictor) against 17–18 ms in llama.cpp, i.e. 3× faster per frame, and its code2wav runs in parallel in a second stage, so RTF is 0.10 (CustomVoice) / 0.20 (Base) against 0.28–0.31. Both are far below real time; the difference only matters for concurrency headroom or for smaller GPUs. +- **First audio** favours llama.cpp once reference-audio handling is in the picture: vLLM-Omni spends ≈0.3–0.5 s per request on x-vector extraction (the chunk cadence after the first chunk is normal), while llama.cpp encodes the speaker once at load. Without reference audio (CustomVoice) vLLM-Omni's measured TTFA of 41–54 ms is below llama.cpp's estimated 109–146 ms. Caching the x-vector per speaker on the server would bring the Base path to the same range; that is the obvious next fix on the branch. +- **Startup and footprint** favour llama.cpp by more than an order of magnitude: 3–6 s to load and be ready (per invocation, speaker encoding included) versus 87–150 s (and 355 s with cold compile caches) for vLLM-Omni, whose two stages each carry a vLLM engine process with CUDA-graph capture and a `torch.compile` warm-up, and 36–37 GB of reserved GPU memory (memory fractions, not need) versus a few GB of weights. + +## 4. CPU only (NUMA node 1 of the Xeon 8480C, pinned) + +Full grid in [results_summary.md](experiments/edge_engine_compare/results_summary.md); the WP3 CustomVoice sweep is in [experiments.md §8.4](experiments.md#84-cpu-platform-wp3-and-cpu-thread-sweep). Both engines pinned to the same cores of NUMA node 1 (stage 0 gets 3/4 of the threads in vLLM-Omni). + +| engine | model | threads | TTFA | playback-start | talker ms/frame | vocoder ms/frame | RTF | startup | peak RSS | +|---|---|---|---|---|---|---|---|---|---| +| llama.cpp | 0.6B Base f16 | 8 / 16 / 32 | 443 / 413 / 407 ms (est.) | – | 94 / 97 / 93 | 55 | 1.9 / 2.0 / 2.0 | 3–4 s | not sampled (Q8_0 run: 6.8 GiB) | +| llama.cpp | 0.6B Base Q8_0 | 16 | 368 ms (est.) | – | 94 | 56 | 1.9 | 2.9 s | 6.8 GiB | +| llama.cpp | 0.6B Base Q4_0 | 8 / 16 | 342 / 332 ms (est.) | – | 85 / 87 | 54 | 1.9 / 1.9 | 2.2 s | 6.3 GiB | +| llama.cpp | 1.7B Base f16 | 16 | 541 ms (est.) | – | 114 | 56 | 2.2 | 5.1 s | 12.2 GiB | +| llama.cpp | 1.7B Base Q4_0 | 16 | 407 ms (est.) | – | 91 | 53 | 2.0 | 3.0 s | 8.0 GiB | +| vLLM-Omni CPU platform | 0.6B Base, x-vector | 8 / 16 / 32 | 855 / 581 / 707 ms | 8.4 / 5.1 / 4.6 s | ≈250 / 147 / 136 (from the CustomVoice sweep) | concurrent stage | 2.6 / 1.8 / 1.7 | 102–104 s | 13.2–13.4 GiB | +| **vLLM-Omni CPU platform after the 2026-09-10 CPU changes** ([mobile_htp_simulation.md §3](mobile_htp_simulation.md); faithful fp32 predictor path, the branch default) | 0.6B Base, x-vector | 16 / 32 | 686 / 486 ms (x-vector included) | 3.0 / 2.2 s | 109 / 91 mean (talker 25 / 21 + predictor 69 / 56 + sampler <1) | concurrent stage | **1.27 / 0.99** | ≈100 s | ≈13 GiB | +| same, CustomVoice (no reference audio) | 0.6B | 8 / 16 / 32 | 388 / 224 / 195 ms | 5.7 / 2.6 / 1.9 s | 148 / 93 / 77 | concurrent stage | 1.93 / **1.15 / 0.89** | ≈100 s | ≈13 GiB | +| same with the int8-dynamic predictor (measured, then **rejected**: 22 % top-1 agreement with the fp32 predictor, see §3 of the HTP document) | 0.6B Base x-vector / CustomVoice | 16 / 32 | 696 / 175 ms | 2.6 / 1.7 s | 80 / 59 | concurrent stage | 1.12 / 0.80 | ≈100 s | ≈13 GiB | +| vLLM-Omni CPU platform | 0.6B CustomVoice (WP3 sweep) | 8 / 16 / 32 | 426 / 292 / 252 ms | 9.4 / 5.3 / 5.0 s | 250 / 147 / 136 | concurrent stage | 2.8 / 1.8 / 1.7 | 110–156 s | 13.3 GiB | + +Reading: +- Update 2026-09-10: after the KV-cached code predictor with a compiled shape-static step, the compiled post-step (lm head + top-k sampling + embedding) and the model-side sampler ([mobile_htp_simulation.md](mobile_htp_simulation.md) §3), the branch generates a frame in 77–109 ms and runs at RTF 0.89–1.27 on 16–32 threads (real time at 32 threads), i.e. 1.6–2.2× faster end-to-end than llama.cpp, whose RTF stays 1.9–2.0 because its talker frame (85–97 ms) is followed by a serial ≈55 ms vocoder pass per frame. At 8 threads the two are level (RTF 1.93 vs 1.9), where llama.cpp's Q4_0 talker frame (85 ms) is still shorter than the branch's 148 ms; an int8 predictor would close that gap but the only int8 path available in this torch build (dynamic per-tensor int8) was rejected for fidelity. The paragraphs below describe the state before that change. +- Neither engine reaches real time on this CPU class with the 0.6B model. llama.cpp's talker is faster per frame (85–97 ms vs 136–250 ms; vLLM-Omni's step includes the 15-codebook predictor eagerly on CPU) but llama.cpp then pays ≈55 ms per frame in its non-streamed BigVGAN vocoder, so the end-to-end RTF is the same ≈1.9–2.0 for llama.cpp and 1.7–2.6 for vLLM-Omni (whose decoder runs concurrently in the second stage). +- llama.cpp needs half the RAM (6–8 GiB vs 13 GiB) and starts in 2–5 s instead of ≈100 s. Quantization trims its talker (Q4_0: −10 %) and its footprint but not the f16 vocoder; vLLM-Omni's CPU platform has no quantized talker path on this branch (bf16/AMX), the first thing to add if the CPU tier matters. +- Playback-start latency is the honest streaming metric on CPU: vLLM-Omni streams but underruns (first audio at 0.6–0.9 s, then the player would stall for 4–8 s over the utterance); llama.cpp does not stream at all (the whole utterance is ready after ≈2× its duration). Threads beyond 16 help neither engine. + +## 5. Bottom line for the edge plan + +- On a Jetson/discrete-GPU class device, the branch's vLLM-Omni is the faster steady-state engine (3× per frame) and the better streaming server (measured 41–54 ms first audio, ramped chunks, concurrent decoder stage), but it costs 100 s+ of startup, a ~9 GiB host process and a Python/vLLM stack; llama.cpp starts in seconds, needs one process and a few GB, and is within 3× on generation speed. For an always-on edge server the branch is the right tier; for a device that must start on demand, or has < 8 GB, llama.cpp is. +- On CPU-only devices neither runs Qwen3-TTS in real time; the study's conclusion stands (real-time CPU TTS needs a smaller/quantized talker and a streamed, cheaper vocoder), and llama.cpp's quantized GGUF path is the better starting point there. +- MNN and ExecuTorch numbers are still missing for the reasons above; the exact unblock is `git clone https://github.com/QwenLM/Qwen3-TTS` (then re-run `llmexport.py` with `PYTHONPATH=`) and `pip install executorch` respectively. diff --git a/docs/engines/executorch.md b/docs/engines/executorch.md new file mode 100644 index 000000000..4d38ea700 --- /dev/null +++ b/docs/engines/executorch.md @@ -0,0 +1,724 @@ +# ExecuTorch (pytorch/executorch) — Engine Analysis + +| Field | Value | +|---|---| +| Local path | `/data/zhoutaichang/embedding_infer/executorch` | +| Remote | `https://github.com/pytorch/executorch.git` (origin) | +| Branch | `main` | +| HEAD | `cfc6ecd5166bec70c5fde2bad836fed26018fe52` — "Qualcomm AI Engine Direct - Enable QNN Windows ARM64 build in CI (#22438)", 2026-09-07 23:47 +0800 | +| Declared version | `1.5.0` ([version.txt](../../executorch/version.txt)) | +| Dirty status | clean (`git status --short` empty) | +| Submodules | **all 23 uninitialized** (`git submodule status` shows `-` for every entry), including `extension/llm/tokenizers`, `backends/xnnpack/third-party/{XNNPACK,pthreadpool,cpuinfo,FP16,FXdiv}`, `backends/mlx/third-party/mlx`, `backends/vulkan/third-party/*`, `third-party/{ao,flatbuffers,flatcc,gflags,googletest,json,pocketfft,pybind11,ios-cmake,prelude}`, `kernels/optimized/third-party/eigen`, `shim`, `backends/cadence/utils/FACTO` | +| Analysis date | 2026-09-08 (read-only; no builds, installs or GPU use) | +| Repo guidance followed | [CLAUDE.md](../../executorch/CLAUDE.md) (AGENTS.md is a symlink to it), `.claude/*.md` reference docs, `.wiki/index.md` | + +Evidence labels: **[Verified]** read in code/docs at this HEAD; **[Inferred]** reasoned from structure; **[Proposal]** design suggestion; **[Unknown]** not determinable from the tree. Repo-published numbers are labelled "repo-reported, not reproduced". + +--- + +## 0. Summary + +- **What it is** [Verified]: ExecuTorch is an *ahead-of-time (AOT) compiler + minimal C++ execution runtime* for PyTorch models. Python side (`exir/`) lowers `torch.export` programs through Edge IR, partitions subgraphs to hardware "delegates" (`to_backend`), plans memory statically, and serializes a FlatBuffer `.pte` ([schema/program.fbs](../../executorch/schema/program.fbs)). The C++ runtime ([runtime/executor/method.cpp](../../executorch/runtime/executor/method.cpp)) walks a linear instruction list per method. +- **Execution runtime, not a serving stack** [Verified]: the core is a single-`Method`, single-thread, sequential interpreter (`Method::execute` loops chains → instructions; comment says "Chains are executed sequentially today"). There is no request queue, batcher or server in `runtime/`. +- **Serving-adjacent layers exist above the core** [Verified]: `extension/llm/runner/` (TextLLMRunner / MultimodalRunner with token callbacks), an experimental `extension/llm/batching/` (`DecodeFirstScheduler`, `Runner`, `Session`, `GenerationHandle` with cancel) and `examples/llm_server/` (OpenAI-compatible Python control plane over JSONL worker binaries). No in-tree class implements `batching::Executor` outside its tests. +- **Backends in tree** [Verified]: XNNPACK, Vulkan, WebGPU, Qualcomm QNN (HTP/GPU/LPAI), MediaTek Neuron, Samsung Exynos (ENN), Arm (Ethos-U via TOSA/Vela, VGF, Cortex-M CMSIS-NN), Cadence DSP, NXP Neutron, OpenVINO, Apple CoreML, Apple Metal (experimental), Apple "Core AI" (under construction), MLX (experimental), CUDA (AOTInductor + Triton; experimental ROCm), a "native" portable delegate, and the AOTI common library. **No `backends/apple/mps` directory exists at this HEAD.** +- **Device model** [Verified]: backend selection happens entirely at export (partitioner). At runtime `BackendDelegate::Init` looks up the delegate by string id in a 16-slot static registry (`kMaxRegisteredBackends = 16`, [runtime/backend/interface.cpp](../../executorch/runtime/backend/interface.cpp#L20)) and calls `is_available()`. There is no runtime device discovery/selection. +- **Memory** [Verified]: static memory planning in Python (`greedy` default, optional banked planning) produces `non_const_buffer_sizes`; the runtime `MemoryManager` = method allocator + `HierarchicalAllocator` (planned arenas, ≤16 spans) + temp allocator. A new `Device{CPU,CUDA}` concept and `DeviceAllocator` registry let planned buffers live in CUDA memory with explicit `_h2d_copy/_d2h_copy` ops inserted at delegate boundaries. +- **Kernels** [Verified]: portable (209 ops in `functions.yaml`), optimized (23), quantized (19), ATen wrapper (225) plus LLM custom ops (`custom_sdpa`, `update_cache`, `quantized_moe_ffn`, `gated_delta_rule`, ...). Kernel table is a static array (default 250 ops × 8 kernels), selective build generates `selected_max_kernel_num.h`. +- **Quantization** [Verified]: torchao source transforms (`8da4w`, `int8`, `4w`, `torchao:8da*`, `torchao:fpa*`) and PT2E quantizers per backend (XNNPACK, QNN, CoreML, Vulkan, OpenVINO, TOSA/Ethos-U/VGF). CUDA backend has int4/5/6/8 packed GEMM shims; MLX has int2/4/8 weight quant. +- **LLM decode loop** [Verified]: `TextPrefiller::prefill` (chunked) → `TextTokenGenerator::generate` → per-token `TextDecoderRunner::step` → `Module::execute("forward"|"decode")` → `Sampler` → tokenizer `decode` → `token_callback`. KV cache is a mutable buffer inside the `.pte` (static, `[B,H,max_context,D]` layout), updated by `update_cache`/`custom_sdpa`. +- **TTS in tree** [Verified]: `examples/models/voxtral_tts` (LM + flow head + codec decoder, 5 methods, CPU/XNNPACK and CUDA, streaming codec chunks with `AudioChunkCallback`) and `examples/models/supertonic` (MLX, 4 methods incl. `vocoder`, dynamic FP16). Mimi codec on QNN under `examples/qualcomm/oss_scripts/moshi`. Audio front end: `extension/audio/mel_spectrogram.py` (exportable Whisper mel), `extension/asr/runner` (seq2seq + transducer), `extension/llm/runner/audio.h`. +- **VLA in tree** [Verified]: none. No OpenVLA/pi0/LeRobot/diffusion-policy/proprioceptive references anywhere. Vision encoders exist for VLMs (LLaVA, Llama 3.2 Vision, Gemma3, SmolVLM, InternVL3) via `MultimodalRunner` with `vision_encoder`/`token_embedding`/`text_decoder` method naming. +- **Edge profile** [Verified]: core runtime CI thresholds ~45–52 KB stripped (Linux gcc/clang size_test), ~118 KB bare-metal Cortex-M / ~136 KB Zephyr all-in; CMake presets for android-arm64-v8a, ios, macos, linux, windows, arm-baremetal, arm-ethosu-linux, esp-baremetal, riscv64-linux, zephyr, llm-release-{cuda,rocm,metal}, mlx-release. Android AAR (Maven), iOS xcframeworks (SwiftPM), Python wheels incl. aarch64 CUDA wheels. +- **Jetson** [Unknown]: no explicit Jetson/Orin documentation; an aarch64-Linux CUDA wheel workflow exists ([.github/workflows/build-wheels-cuda-aarch64-linux.yml](../../executorch/.github/workflows/build-wheels-cuda-aarch64-linux.yml)), so aarch64+CUDA is at least built. +- **Tokenizers** [Verified]: C++ tokenizers (HF JSON, Tiktoken, SentencePiece, Llama2c, Tekken) live in the `extension/llm/tokenizers` submodule, which is **not checked out here**; only the `.claude/tokenizers.md` summary and call sites could be inspected. + +--- + +## 1. Architecture overview + +```mermaid +graph TD + subgraph AOT["AOT (Python, exir/)"] + TE["torch.export → ExportedProgram"] --> TOEDGE["to_edge / to_edge_transform_and_lower
EdgeProgramManager"] + TOEDGE --> PART["Partitioner(s) → to_backend()
BackendDetails.preprocess → delegate blob"] + PART --> PASSES["edge_to_executorch_passes
ConstraintBasedSymShapeEvalPass, reinplace, PropagateDevicePass"] + PASSES --> MP["MemoryPlanningPass (greedy / banked)"] + MP --> EMIT["_Emitter → ExecutionPlan(FlatBuffer)"] + EMIT --> PTE[".pte (+ .ptd via flat_tensor)"] + end + subgraph RT["Runtime (C++, runtime/)"] + PTE --> DL["DataLoader (file/mmap/buffer)"] + DL --> PROG["Program::load → Program::load_method"] + PROG --> METH["Method::init
parse_values / resolve delegates / resolve_operator"] + METH --> EXEC["Method::execute → execute_instruction
KernelCall | DelegateCall | JumpFalse | Move | Free"] + EXEC --> KREG["Kernel registry
portable / optimized / quantized / custom"] + EXEC --> BREG["Backend registry (≤16)
BackendInterface::init/execute"] + MM["MemoryManager
method alloc + HierarchicalAllocator + temp"] --> METH + DEV["DeviceAllocatorRegistry
CPU / CUDA"] --> MM + PAL["Platform PAL (log, clock, alloc, abort)"] --> EXEC + end + subgraph EXT["Extensions"] + MOD["extension/module::Module"] --> PROG + LLM["extension/llm/runner
TextLLMRunner / MultimodalRunner"] --> MOD + BATCH["extension/llm/batching
DecodeFirstScheduler / Runner"] -.-> LLM + ASR["extension/asr/runner"] --> MOD + THR["extension/threadpool (pthreadpool+cpuinfo)"] --> KREG + THR --> BREG + BIND["android JNI / apple Swift / pybindings / wasm"] --> MOD + SRV["examples/llm_server (OpenAI HTTP)"] -.-> LLM + end + subgraph BE["Delegates (backends/)"] + BREG --> XNN["XNNPACK"] & VK["Vulkan"] & QNN["Qualcomm QNN"] & CML["CoreML"] & MLX["MLX"] & CUDA["CUDA (AOTI)"] & ARM["Arm Ethos-U / VGF"] & OTH["MediaTek / Samsung / NXP / OpenVINO / Cadence / WebGPU / Metal / native"] + end +``` + +**Key modules** + +| Module | Path | Responsibility | Key symbols | +|---|---|---|---| +| Export IR | [exir/program/_program.py](../../executorch/exir/program/_program.py) | `to_edge`, `to_edge_transform_and_lower`, `EdgeProgramManager.to_backend/to_executorch`, `ExecutorchProgramManager.buffer/save` | `to_edge` (L1414), `EdgeProgramManager` (L1496), `ExecutorchProgramManager` (L1823) | +| Backend API | [exir/backend/backend_api.py](../../executorch/exir/backend/backend_api.py), [exir/backend/partitioner.py](../../executorch/exir/backend/partitioner.py), [exir/backend/backend_details.py](../../executorch/exir/backend/backend_details.py) | Partition + preprocess into delegate blobs | `to_backend` (singledispatch), `Partitioner.partition`, `BackendDetails.preprocess/preprocess_multimethod`, `PreprocessResult` | +| Memory planning | [exir/memory_planning.py](../../executorch/exir/memory_planning.py), [exir/banked_memory_planning.py](../../executorch/exir/banked_memory_planning.py) | Lifetime analysis, arena assignment | `greedy`, `naive`, `MemoryPlanningAlgorithmSuite`, `_partition_specs_by_device`, `TargetMemoryMap` | +| Config | [exir/capture/_config.py](../../executorch/exir/capture/_config.py) | Export/backend knobs | `EdgeCompileConfig`, `ExecutorchBackendConfig` | +| Emitter | [exir/emit/_emitter.py](../../executorch/exir/emit/_emitter.py) | FX → FlatBuffer instructions | `_Emitter`, `_TopLevelEmitter` | +| Schema | [schema/program.fbs](../../executorch/schema/program.fbs) | `.pte` file format | `Program`, `ExecutionPlan`, `Chain`, `KernelCall`, `DelegateCall`, `NonConstBufferDevice` | +| Program/Method | [runtime/executor/program.h](../../executorch/runtime/executor/program.h), [runtime/executor/method.cpp](../../executorch/runtime/executor/method.cpp) | Load, init, execute | `Program::load/load_method`, `Method::init/execute/step/set_input/get_outputs`, `BackendDelegate` | +| Memory | [runtime/executor/memory_manager.h](../../executorch/runtime/executor/memory_manager.h), [runtime/core/hierarchical_allocator.h](../../executorch/runtime/core/hierarchical_allocator.h), [runtime/core/memory_allocator.h](../../executorch/runtime/core/memory_allocator.h) | Arena allocators | `MemoryManager`, `HierarchicalAllocator::get_offset_address`, `MemoryAllocator` | +| Device memory | [runtime/core/device_allocator.h](../../executorch/runtime/core/device_allocator.h), [runtime/core/device_memory_buffer.h](../../executorch/runtime/core/device_memory_buffer.h), [runtime/core/portable_type/device.h](../../executorch/runtime/core/portable_type/device.h) | Non-CPU planned buffers | `DeviceAllocator`, `DeviceAllocatorRegistry`, `DeviceMemoryBuffer`, `Device{CPU,CUDA}` | +| Backend registry | [runtime/backend/interface.h](../../executorch/runtime/backend/interface.h) | Delegate ABI | `BackendInterface::{is_available,init,execute,set_option,get_option,destroy}`, `register_backend` | +| Kernel registry | [runtime/kernel/operator_registry.h](../../executorch/runtime/kernel/operator_registry.h) | Op → kernel resolution | `Kernel`, `KernelKey`, `register_kernels`, `get_op_function_from_registry` | +| PAL | [runtime/platform/platform.h](../../executorch/runtime/platform/platform.h) | OS abstraction | `et_pal_init/abort/current_ticks/emit_log_message/allocate/free`, `PalImpl` | +| Kernels | [kernels/portable/functions.yaml](../../executorch/kernels/portable/functions.yaml), [kernels/optimized/optimized.yaml](../../executorch/kernels/optimized/optimized.yaml), [kernels/quantized/quantized.yaml](../../executorch/kernels/quantized/quantized.yaml) | CPU op libraries | YAML op defs + `kernels/*/cpu/op_*.cpp` | +| Selective build | [tools/cmake/Codegen.cmake](../../executorch/tools/cmake/Codegen.cmake), [codegen/gen.py](../../executorch/codegen/gen.py) | Generate registration for selected ops | `gen_selected_ops`, `EXECUTORCH_SELECT_OPS_*` | +| Module API | [extension/module/module.h](../../executorch/extension/module/module.h) | Convenience wrapper | `Module::{load_method,execute,forward,set_input,set_output,method_meta}`, `LoadMode` | +| LLM runner | [extension/llm/runner/](../../executorch/extension/llm/runner/README.md) | Prefill/decode loop | `IRunner`, `TextLLMRunner`, `TextPrefiller`, `TextDecoderRunner`, `TextTokenGenerator`, `IOManager`, `MultimodalRunner`, `MultimodalPrefiller`, `LLMSession`, `LLMEngine` | +| LLM batching | [extension/llm/batching/](../../executorch/extension/llm/batching/runner.h) | Experimental multi-session scheduler | `Scheduler`, `DecodeFirstScheduler`, `Executor`, `Runner`, `Session`, `GenerationHandle` | +| LLM cache | [extension/llm/cache/](../../executorch/extension/llm/cache/cache.h) | Off-graph KV cache abstractions | `CacheBase`, `SequenceCache`, `CellCache`, `RingPolicy`, `kvcache::update_and_attend` | +| Custom ops | [extension/llm/custom_ops/custom_ops.py](../../executorch/extension/llm/custom_ops/custom_ops.py) | LLM kernels | `sdpa_with_kv_cache`, `custom_sdpa`, `update_cache`, `quantized_moe_ffn`, `gated_delta_rule` | +| Threadpool | [extension/threadpool/threadpool.h](../../executorch/extension/threadpool/threadpool.h), [runtime/kernel/thread_parallel_interface.h](../../executorch/runtime/kernel/thread_parallel_interface.h) | CPU parallelism | `ThreadPool`, `get_threadpool`, `get_pthreadpool`, `parallel_for` | +| Devtools | [devtools/etdump/](../../executorch/devtools/etdump/etdump_flatcc.h), [devtools/inspector/_inspector.py](../../executorch/devtools/inspector/_inspector.py), [devtools/bundled_program/](../../executorch/devtools/bundled_program/bundled_program.h) | Profiling / debugging | ETDump, `Inspector`, bundled program | + +--- + +## 2. Device layer + +### 2.1 Backends actually in tree [Verified] + +Every backend registers itself with `register_backend(...)` into a static 16-entry table ([runtime/backend/interface.cpp](../../executorch/runtime/backend/interface.cpp#L20), `kMaxRegisteredBackends = 16`, with a TODO noting global statics prevent multiple executors). Registration sites: + +| Backend id / dir | Registration | Hardware / OS | Status note | +|---|---|---|---| +| XNNPACK — [backends/xnnpack](../../executorch/backends/xnnpack/README.md) | [XNNPACKBackend.cpp#L272](../../executorch/backends/xnnpack/runtime/XNNPACKBackend.cpp#L272) | ARM64/ARMv7/x86 CPU on Android, iOS, macOS, Linux, Windows ([xnnpack-overview.md](../../executorch/docs/source/backends/xnnpack/xnnpack-overview.md)); KleidiAI via `EXECUTORCH_XNNPACK_ENABLE_KLEIDI` ([CMakeLists.txt#L34](../../executorch/backends/xnnpack/CMakeLists.txt#L34)) | default CPU delegate | +| Vulkan — [backends/vulkan](../../executorch/backends/vulkan/README.md) | [VulkanBackend.cpp#L931](../../executorch/backends/vulkan/runtime/VulkanBackend.cpp#L931) | Android GPUs (Adreno/Mali/PowerVR noted in wiki), desktop experimental | mature on Android | +| WebGPU — [backends/webgpu](../../executorch/backends/webgpu/README.md) | [WebGPUBackend.cpp#L263](../../executorch/backends/webgpu/runtime/WebGPUBackend.cpp#L263) | Dawn/Tint WGSL; browser via Emscripten, native Metal/Vulkan | under active development | +| Qualcomm QNN — [backends/qualcomm](../../executorch/backends/qualcomm/README.md) | [QnnExecuTorchBackend.cpp#L394](../../executorch/backends/qualcomm/runtime/QnnExecuTorchBackend.cpp#L394) | HTP / GPU / LPAI; 38 chipset enum entries in [qc_schema.py](../../executorch/backends/qualcomm/serialization/qc_schema.py) (`QcomChipset`) | no BC guarantees stated | +| MediaTek Neuron — [backends/mediatek](../../executorch/backends/mediatek/README.md) | [NeuronBackend.cpp#L261](../../executorch/backends/mediatek/runtime/NeuronBackend.cpp#L261) | Dimensity 9300/9400 NPU | | +| Samsung Exynos — [backends/samsung](../../executorch/backends/samsung/README.md) | [enn_backend.cpp#L96](../../executorch/backends/samsung/runtime/enn_backend.cpp#L96) | Exynos 2500 NPU/DSP (`EXECUTORCH_BUILD_ENN`) | | +| Arm Ethos-U — [backends/arm](../../executorch/backends/arm/README.md) | [EthosUBackend.cpp#L451](../../executorch/backends/arm/runtime/EthosUBackend.cpp#L451) | Ethos-U55/65/85 via TOSA → Vela | bare-metal | +| Arm VGF — same dir | [VGFBackend.cpp#L803](../../executorch/backends/arm/runtime/VGFBackend.cpp#L803) | Vulkan ML extensions (SPIR-V ML) | | +| Arm Cortex-M — [backends/cortex_m](../../executorch/backends/cortex_m/README.md) | not a delegate: an operator dialect (`cortex_m::*`) on CMSIS-NN with portable fallback | Cortex-M MCUs | Beta | +| Cadence — [backends/cadence](../../executorch/backends/cadence/README.md) | no delegation; custom ops for HiFi4/FusionG3/Vision DSP (see wiki "No-delegation architecture") | Xtensa DSP | | +| NXP Neutron — [backends/nxp](../../executorch/backends/nxp/README.md) | [NeutronBackend.cpp#L676](../../executorch/backends/nxp/runtime/NeutronBackend.cpp#L676) | eIQ Neutron NPU (i.MX) | | +| OpenVINO — [backends/openvino](../../executorch/backends/openvino/README.md) | [OpenvinoBackend.cpp#L419](../../executorch/backends/openvino/runtime/OpenvinoBackend.cpp#L419) | Intel CPU / iGPU / dGPU / NPU | | +| Apple CoreML — [backends/apple/coreml](../../executorch/backends/apple/coreml/README.md) | [coreml_backend_delegate.mm#L438](../../executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm#L438) | iOS/macOS ANE/GPU/CPU (compute units via compile spec) | | +| Apple Metal — [backends/apple/metal](../../executorch/backends/apple/metal/README.md) | [metal_backend.cpp#L696](../../executorch/backends/apple/metal/runtime/metal_backend.cpp#L696) | AOTI-driven Metal | "EXPERIMENTAL" | +| Apple Core AI — [backends/apple/coreai](../../executorch/backends/apple/coreai/README.md) | Python only at this HEAD | | "Under construction — not for use" | +| MLX — [backends/mlx](../../executorch/backends/mlx/README.md) | [MLXBackend.cpp#L570](../../executorch/backends/mlx/runtime/MLXBackend.cpp#L570) | Apple Silicon GPU, macOS ≥ 14, iOS ≥ 17 experimental ([mlx-overview.md](../../executorch/docs/source/backends/mlx/mlx-overview.md)) | experimental | +| CUDA — [backends/cuda](../../executorch/backends/cuda/cuda_backend.py) | [cuda_backend.cpp#L1429](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L1429) | NVIDIA Linux/Windows via AOTInductor + Triton; ROCm experimental ([rocm.md](../../executorch/backends/cuda/rocm.md)) | | +| AOTI common — [backends/aoti](../../executorch/backends/aoti/README.md) | library, not a backend | shared shims for CUDA/Metal | | +| native — [backends/native](../../executorch/backends/native/preprocess.py) | `NativeBackend` serializes an fx subgraph to a generic flatbuffer executed by [backends/native/runtime/Program.h](../../executorch/backends/native/runtime/Program.h) | portable | new | +| example/test — [backends/example](../../executorch/backends/example/README.md), [backends/test](../../executorch/backends/test/README.md) | reference delegate + test harness | | | + +Reference table also in [.claude/backends.md](../../executorch/.claude/backends.md) and [docs/source/backends-overview.md](../../executorch/docs/source/backends-overview.md). + +### 2.2 Device discovery / selection [Verified] + +- **AOT**: the user chooses the target by passing a `Partitioner` to `to_backend()`/`to_edge_transform_and_lower()`. Multiple partitioners are applied in priority order; unmatched ops stay as `KernelCall`s for the CPU kernel library ([.claude/backends.md](../../executorch/.claude/backends.md)). +- **Runtime**: `BackendDelegate::Init` ([method.cpp#L61](../../executorch/runtime/executor/method.cpp#L61)) does `get_backend_class(backend_id)` then `backend->is_available()` and fails with `NotFound` if either check fails. There is no fallback to another backend at runtime and no enumeration of physical devices. Backend-specific selection (e.g., CoreML compute units, QNN SoC/HTP arch, CUDA graph enable) is carried in `CompileSpec`s embedded in the `.pte` and parsed in `init` (e.g., [coreml_backend_delegate.mm#L278](../../executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm#L278), `kEnableCudaGraphForMethod` in [cuda_backend.cpp#L85](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L85)). +- Runtime knobs can be pushed to a backend by name with `set_option(backend_name, Span)` ([interface.h](../../executorch/runtime/backend/interface.h)); a `BackendOptionsMap` can be passed at `load_method` time. + +### 2.3 Memory allocation / pools [Verified] + +- `MemoryManager(method_allocator, planned_memory, temp_allocator)` ([memory_manager.h](../../executorch/runtime/executor/memory_manager.h)); method and temp allocator must differ. `planned_memory` is a `HierarchicalAllocator` over `Span>` — one span per `mem_id`, hard cap `kSpanArraySize = 16` for the deprecated constructor; `get_offset_address(memory_id, offset, size)` with overflow checks ([hierarchical_allocator.h](../../executorch/runtime/core/hierarchical_allocator.h)). +- Sizes come from `ExecutionPlan.non_const_buffer_sizes` ([program.fbs#L368](../../executorch/schema/program.fbs#L368)); the app must allocate those arenas. `Method::resolve_operator` uses temp allocator (falls back to method allocator) for `TensorMeta` scratch ([method.cpp#L731](../../executorch/runtime/executor/method.cpp#L731)). +- Temp allocator is reset after each instruction and at the start of `execute()` ([method.cpp#L1697](../../executorch/runtime/executor/method.cpp#L1697)) — kernels get per-op scratch. +- Planning algorithms: `greedy` (default, allows overlapping lifetimes), `naive`, `MemoryPlanningAlgorithmSuite` picks the minimum total; `banked_memory_planning.TargetMemoryMap` adds capacity-aware placement across memory banks with spill ([banked_memory_planning.py](../../executorch/exir/banked_memory_planning.py)). Mutable buffers (KV caches) are placeholders that are "never freed" ([memory_planning.py#L492](../../executorch/exir/memory_planning.py#L492)) and are planned only if `alloc_mutable_buffers`. +- Inputs/outputs: if not memory-planned, `Method::set_input` aliases user memory (`share_tensor_data`) instead of copying; otherwise `copy_tensor_data` after `resize_tensor` ([method.cpp#L1146](../../executorch/runtime/executor/method.cpp#L1146)); `set_output_data_ptr` lets the caller supply output buffers. + +### 2.4 Host ↔ device transfers [Verified] + +- `Device{type: CPU|CUDA, index}` ([device.h](../../executorch/runtime/core/portable_type/device.h)); `DeviceAllocator` has `allocate/deallocate/copy_host_to_device/copy_device_to_host`; `DeviceAllocatorRegistry` holds one allocator per device type; `DeviceMemoryBuffer::create(size, type)` RAII ([device_allocator.h](../../executorch/runtime/core/device_allocator.h), [device_memory_buffer.h](../../executorch/runtime/core/device_memory_buffer.h)). +- `ExecutionPlan.non_const_buffer_device: [NonConstBufferDevice]` marks planned arenas on CUDA ([program.fbs#L417](../../executorch/schema/program.fbs#L417)); `MemoryManager::has_device_memory()` exposes this. +- At export, `ExecutorchBackendConfig.enable_non_cpu_memory_planning=True` (default) partitions specs by device (`_partition_specs_by_device`) and `PropagateDevicePass` inserts `_h2d_copy`/`_d2h_copy` ops at delegate boundaries ([_config.py#L126](../../executorch/exir/capture/_config.py#L126), [propagate_device_pass.py](../../executorch/exir/passes/propagate_device_pass.py), [_device_copy_ops_registry.py#L27](../../executorch/exir/passes/_device_copy_ops_registry.py#L27)). +- CUDA backend auto-registers `CudaAllocator` ([cuda_backend.cpp#L1431](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L1431), [cuda_allocator.h](../../executorch/backends/cuda/runtime/cuda_allocator.h)) and uses `cudaMemcpyAsync` device-to-device for CUDA-graph static I/O staging ([cuda_backend.cpp#L574](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L574)). +- Other delegates copy internally (XNNPACK: `xnn_setup_runtime_v2` on external tensors; Vulkan: `maybe_resize_input`/staging in [VulkanBackend.cpp#L529](../../executorch/backends/vulkan/runtime/VulkanBackend.cpp#L529); QNN: `RegisterMem`/`PreRegisterMem` shared buffers and rpcmem in [QnnManager.h#L89](../../executorch/backends/qualcomm/runtime/QnnManager.h#L89), [rpc_mem.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/rpc_mem.h)). + +### 2.5 Synchronization / streams [Verified] + +- Core runtime is synchronous: every `KernelCall`/`DelegateCall` returns before the next instruction. No stream abstraction in `runtime/`. +- CUDA backend: optional CUDA-graph capture per method (`CudaGraphPhase::{Warmup,Replay,Disabled}`, `kCudaGraphWarmupSteps`, `cudaGraphLaunch` on a stream — [cuda_backend.cpp#L434](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L434)); enable with compile-spec `enable_cuda_graph_for_method`. Unavailable on ROCm ([rocm.md](../../executorch/backends/cuda/rocm.md)). +- Vulkan: `ComputeGraph::submit_current_cmd[_and_wait]`, `submit_deferred_cmds_and_wait`, `prepack`, `optional_warmup_execute`, `propagate_resize` ([ComputeGraph.h#L1103](../../executorch/backends/vulkan/runtime/graph/ComputeGraph.h#L1103)). +- MLX: `async_eval` with keep-alive of input data ([MLXBackend.cpp#L94](../../executorch/backends/mlx/runtime/MLXBackend.cpp#L94)). + +### 2.6 Portability [Verified] + +- PAL functions are weak symbols (`ET_INTERNAL_PLATFORM_WEAKNESS`) or a `PalImpl` table ([platform.h](../../executorch/runtime/platform/platform.h)); defaults in `runtime/platform/default`. Examples of custom PALs: Arduino ([examples/arduino/arduino_pal.cpp](../../executorch/examples/arduino/arduino_pal.cpp)), Zephyr ([zephyr/](../../executorch/zephyr)), CUDA ([backends/cuda/runtime/platform/platform.cpp](../../executorch/backends/cuda/runtime/platform/platform.cpp)). +- Build presets ([CMakePresets.json](../../executorch/CMakePresets.json)): `android-arm64-v8a`, `android-x86_64`, `macos`, `ios`, `ios-simulator`, `linux`, `windows`, `pybind`, `zephyr`, `arm-baremetal`, `arm-ethosu-linux`, `esp-baremetal`, `riscv64-linux`, `llm`, `llm-release[-cuda|-rocm|-metal]`, `llm-debug[-cuda|-metal|-vulkan]`, `mlx[-release|-debug]`, `profiling`. +- Host OS for building: Linux x86_64, macOS x86_64/ARM64, Windows 10+ (VS2022 + Clang-CL) or WSL ([using-executorch-building-from-source.md](../../executorch/docs/source/using-executorch-building-from-source.md)). + +--- + +## 3. Kernel layer + +### 3.1 Operator set & registration [Verified] + +- Op definitions are YAML (`- op:`/`- func:` entries): portable 209 ([functions.yaml](../../executorch/kernels/portable/functions.yaml)) + 2 custom ([custom_ops.yaml](../../executorch/kernels/portable/custom_ops.yaml)); optimized 23 ([optimized.yaml](../../executorch/kernels/optimized/optimized.yaml) — `op_add/bmm/div/elu/exp/fft_*/gelu/grid_sampler_2d/le/linear/log_softmax/mm/mul/native_layer_norm/sub/sum/where`); quantized 19 ([quantized.yaml](../../executorch/kernels/quantized/quantized.yaml)); ATen wrapper 225 ([kernels/aten/functions.yaml](../../executorch/kernels/aten/functions.yaml)); prim ops (`et_copy_index`, `et_view`, sym ops) in [kernels/prim_ops/register_prim_ops.cpp](../../executorch/kernels/prim_ops/register_prim_ops.cpp). +- Codegen ([codegen/gen.py](../../executorch/codegen/gen.py), [codegen/templates](../../executorch/codegen/templates)) emits `Kernel{name, KernelKey, OpFunction}` arrays registered via `register_kernels(Span)` ([operator_registry.h#L179](../../executorch/runtime/kernel/operator_registry.h#L179)). `KernelKey` encodes dtype/dim-order per tensor arg (`kKernelKeyBufSize = 659`); a null key is the fallback kernel. +- Table capacity: `MAX_KERNEL_NUM` → `EXECUTORCH_SELECTED_MAX_KERNEL_NUM` (generated by selective build) → default `250 ops × 8 kernels` ([operator_registry.cpp#L30](../../executorch/runtime/kernel/operator_registry.cpp#L30)). Duplicate registration aborts (see [.claude/faq.md](../../executorch/.claude/faq.md)). +- Portable kernels must use ExecuTorch types (`executorch::aten::Tensor`, no `at::Tensor`) — [kernels/README.md](../../executorch/kernels/README.md). + +### 3.2 Dispatch [Verified] + +- At `Method::init`, each `KernelCall` instruction is resolved once: `resolve_operator` builds `TensorMeta` for the args and calls `get_op_function_from_registry(name, meta_list)`; the resulting `OpFunction` pointer is stored in `chain.kernels_[instr]` ([method.cpp#L731](../../executorch/runtime/executor/method.cpp#L731), [#L1040](../../executorch/runtime/executor/method.cpp#L1040)). +- At `execute_instruction`, `KernelCall` invokes `chain.kernels_[idx](context, args)` with a `KernelRuntimeContext(event_tracer, temp_allocator)`; `DelegateCall` invokes `delegates_[idx].Execute(BackendExecutionContext, args)`; `JumpFalseCall`, `MoveCall`, `FreeCall` implement control flow, aliasing and explicit frees ([method.cpp#L1450](../../executorch/runtime/executor/method.cpp#L1450)). +- Routing to a backend is therefore decided *entirely at export*: an op either became part of a delegate blob (`executorch_call_delegate` → `DelegateCall`) or stays a CPU `KernelCall`. There is no runtime per-op dispatcher across backends. +- When the `Method` is loaded with an explicit `kernel_registry` span, resolution uses that list instead of the global table ([method.h](../../executorch/runtime/executor/method.h) `Span kernel_registry_`). + +### 3.3 Quantization formats & where dequant happens [Verified] + +- **LLM source transforms** ([quantize.py](../../executorch/examples/models/llama/source_transformation/quantize.py)): `int8` (weight-only), `8da4w`/`8da8w` (int8 dynamic act + int4/int8 weight, torchao `Int8DynamicActivationIntxWeightConfig`), `4w`, `torchao:8da*` and `torchao:fpa*` low-bit kernels for Arm, embedding quant `4,32` etc. Config in [llm_config.py](../../executorch/extension/llm/export/config/llm_config.py) (`QuantizationConfig.qmode`, `pt2e_quantize`, `use_qat`, `use_spin_quant`, `preq_mode` values `8da4w` and `8da4w_output_8da8w`). SpinQuant R1/R2 rotations: [apply_spin_quant_r1_r2.py](../../executorch/examples/models/llama/source_transformation/apply_spin_quant_r1_r2.py). +- **KV-cache quant**: `QuantizedKVCache` (int8 per-token, `QuantizedCacheType`), `StaticQuantizedKVCache` with calibration, `QuantizedRingKVCache`, `CustomKVCacheWithAttentionSink` ([custom_kv_cache.py](../../executorch/examples/models/llama/source_transformation/custom_kv_cache.py)); `QuantizedSDPA` consumes int8 caches ([sdpa.py#L104](../../executorch/examples/models/llama/source_transformation/sdpa.py#L104)). Dequant happens inside `custom_quantized_sdpa`/XNNPACK kernels at runtime. +- **PT2E quantizers** ([quantizer_lib.py](../../executorch/extension/llm/export/quantizer_lib.py)): XNNPACK ([xnnpack_quantizer.py](../../executorch/backends/xnnpack/quantizer/xnnpack_quantizer.py)), QNN (`get_qnn_quantizer`, 8a8w/16a4w/16a8w — see wiki [recipes.md](../../executorch/.wiki/quantization/recipes.md)), CoreML, Vulkan, OpenVINO, TOSA/Ethos-U/VGF, Cortex-M `CortexMQuantizer`. Q/DQ nodes are fused into the delegate blob by the backend's preprocess; ops that stay on CPU run `quantized_decomposed::*` kernels from [kernels/quantized](../../executorch/kernels/quantized/quantized.yaml) (see `quant_fusion_pass.py` in [exir/passes](../../executorch/exir/passes/quant_fusion_pass.py)). +- **GPU**: CUDA int4 tile-packed (`tile_packed_to_4d`, `_weight_int4pack_mm`) plus int5/int6/int8 planar GEMM shims ([backends/cuda/runtime/shims](../../executorch/backends/cuda/runtime/shims/int4mm.cuh), [quantize_op_dispatch](../../executorch/backends/cuda/quantize_op_dispatch/int4_dispatch.py)); MLX INT2/4/8 weight quant via torchao ([mlx-overview.md](../../executorch/docs/source/backends/mlx/mlx-overview.md)); WebGPU 4-bit weight-only, 8da4w, q8ta ([webgpu README](../../executorch/backends/webgpu/README.md)). Also `extension/llm/export/{int4,mx,nvfp4,gguf}.py` for GGUF loading, MX and NVFP4 formats. + +### 3.4 Compilation / JIT / graph capture [Verified] + +- Everything is AOT: `torch.export` → `to_edge` (ATen→Edge dialect, decompositions, dim-order) → partition/preprocess → `to_executorch` (passes incl. `ConstraintBasedSymShapeEvalPass`, `reinplace`, `MemoryPlanningPass`, `ToOutVarPass`) → emit. No JIT at runtime. +- Delegate compilers: XNNPACK builds an `xnn_subgraph` at `init` (`xnn_create_runtime_v4` with pthreadpool, weights cache, workspace — [XNNCompiler.cpp#L2296](../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp#L2296)); Vulkan builds `ComputeGraph` from a flatbuffer, prepacks and optionally warms up ([VulkanBackend.cpp#L676](../../executorch/backends/vulkan/runtime/VulkanBackend.cpp#L676)); QNN loads precompiled context binaries (or online-prepare) via `QnnManager::Compile` ([QnnManager.h#L85](../../executorch/backends/qualcomm/runtime/QnnManager.h#L85)); CoreML compiles/caches `.mlmodelc` at first load; CUDA `dlopen`s the AOTInductor `.so` from the `.ptd` and calls `AOTInductorModelContainerCreateWithDevice/Run` ([cuda_backend.cpp#L164](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L164)); MLX interprets its own flatbuffer bytecode ([backends/mlx/serialization](../../executorch/backends/mlx/serialization)). + +### 3.5 Fallback path [Verified] + +- Unsupported ops simply remain `KernelCall`s to the portable/optimized CPU kernels ([backends-overview.md](../../executorch/docs/source/backends-overview.md): "Operators not supported by the delegate are executed using the portable CPU fallback"). If the CPU kernel is also missing (selective build), `Method::init` fails with `OperatorMissing` (0x14) ([.claude/faq.md](../../executorch/.claude/faq.md)). Cortex-M's dialect explicitly relies on portable-ops fallback ([cortex_m/README.md](../../executorch/backends/cortex_m/README.md)). +- `extension/llm/custom_ops/op_fallback.cpp` provides an AOT-side fallback op ([op_fallback.py](../../executorch/extension/llm/custom_ops/op_fallback.py)). + +### 3.6 Extension points [Verified] + +- **Custom op**: define via `torch.library` (see [custom_ops.py](../../executorch/extension/llm/custom_ops/custom_ops.py)), implement an `out`-variant C++ kernel using ExecuTorch tensors, register with `EXECUTORCH_LIBRARY` / YAML + codegen (`gen_selected_ops`, [Codegen.cmake](../../executorch/tools/cmake/Codegen.cmake)); docs [kernel-library-custom-aten-kernel.md](../../executorch/docs/source/kernel-library-custom-aten-kernel.md), [examples/portable/custom_ops](../../executorch/examples/portable/custom_ops). +- **Custom backend**: Python `BackendDetails.preprocess(edge_program, compile_specs) -> PreprocessResult` + a `Partitioner` returning `PartitionResult` ([backend_details.py#L53](../../executorch/exir/backend/backend_details.py#L53), [partitioner.py#L37](../../executorch/exir/backend/partitioner.py#L37)); C++ `BackendInterface` subclass + `static auto _ = register_backend({"Id", &instance})`; template in [docs/source/backends/template](../../executorch/docs/source/backends/template) and the [backends/example](../../executorch/backends/example/README.md) reference delegate. Multi-method delegates via `preprocess_multimethod` and `MethodProgramsPartitionerSpec` ([backend_api.py#L664](../../executorch/exir/backend/backend_api.py#L664)). +- **Custom memory planning**: pass your own `MemoryPlanningPass`/algorithm list or custom `mem_id` pools ([compiler-memory-planning.md](../../executorch/docs/source/compiler-memory-planning.md)). +- **Custom passes**: `EdgeProgramManager.transform(passes)` and `ExecutorchBackendConfig.passes` ([compiler-custom-compiler-passes.md](../../executorch/docs/source/compiler-custom-compiler-passes.md)). + +--- + +## 4. Model runner + +### 4.1 Loading / export / conversion pipeline [Verified] + +- Formats: `.pte` (FlatBuffer program with inline or segmented constants and delegate blobs; extended header in [schema/extended_header.h](../../executorch/schema/extended_header.h); [pte-file-format.md](../../executorch/docs/source/pte-file-format.md)); `.ptd` external tensor files (`extension/flat_tensor`, `FlatTensorDataMap`; [ptd-file-format.md](../../executorch/docs/source/ptd-file-format.md)) used for external constants (`ExecutorchBackendConfig.external_constants`), CUDA AOTI blobs (`aoti_cuda_blob.ptd`), Whisper weights, LoRA adapters. +- Export API: `to_edge_transform_and_lower(ep, partitioner=[...], transform_passes=..., compile_config=EdgeCompileConfig)` then `.to_executorch(ExecutorchBackendConfig)` and `.buffer`/`.save()` ([_program.py#L1280](../../executorch/exir/program/_program.py#L1280)). Also `executorch.export` recipe API ([export/](../../executorch/export)). +- LLM export: `python -m executorch.extension.llm.export.export_llm` (Hydra/YAML `LlmConfig`: `base`, `model`, `export`, `quantization`, `backend`, `debug`) → [export_llm.py](../../executorch/extension/llm/export/export_llm.py) → [export_llama_lib.py](../../executorch/examples/models/llama/export_llama_lib.py) → `LLMEdgeManager` in [builder.py](../../executorch/extension/llm/export/builder.py). Model classes in `ModelType` (stories110m, llama2/3/3_1/3_2, llama3_2_vision, static_llama, qwen2_5*, qwen3*, qwen3_5*, phi_4_mini, smollm2, lfm2*). HF models via *optimum-executorch* (out of tree; [export-llm-optimum.md](../../executorch/docs/source/llm/export-llm-optimum.md)). QNN LLMs use a separate path [examples/qualcomm/oss_scripts/llama/llama.py](../../executorch/examples/qualcomm/oss_scripts/llama/llama.py) (wiki warns against `export_llama --qnn`, [model-specific.md](../../executorch/.wiki/export/model-specific.md)). +- Loading: `DataLoader` variants (`FileDataLoader`, `MmapDataLoader` with mlock/madvise, `BufferDataLoader`, `FileDescriptorDataLoader` — [extension/data_loader](../../executorch/extension/data_loader/mmap_data_loader.h)); `Module::LoadMode {File, Mmap, MmapUseMlock, MmapUseMlockIgnoreErrors, MmapUseMadvise}` ([module.h#L50](../../executorch/extension/module/module.h#L50)); LLM runner default `MmapUseMlockIgnoreErrors` ([llm_runner_helper.h#L106](../../executorch/extension/llm/runner/llm_runner_helper.h#L106)). + +### 4.2 Execution lifecycle [Verified] + +1. `Program::load(loader, Verification)` → header check, optional flatbuffer verification ([program.h#L97](../../executorch/runtime/executor/program.h#L97)). +2. `Program::load_method(name, MemoryManager*, EventTracer*, NamedDataMap*)` → `Method::init`: `parse_values` (materialize `values_` table, tensors pointing into planned arenas / constant segments / external `.ptd` data), allocate and `BackendDelegate::Init` each delegate (calls `backend->init` → `DelegateHandle*`), `resolve_operator` for every `KernelCall` ([method.cpp#L883](../../executorch/runtime/executor/method.cpp#L883)). +3. `Method::set_input(s)` (resize + copy or alias) → `Method::execute()` (checks all inputs set, resets temp allocator, sequentially runs all chains/instructions, guards against >`ET_MAX_INSTRUCTIONS`=10 M, then `reset_execution`) → `Method::get_outputs` (shallow copies) ([method.cpp#L1697](../../executorch/runtime/executor/method.cpp#L1697)). +4. Alternative: `Method::step()` executes one instruction (experimental; used by devtools/debuggers), `reset_execution()` after `EndOfMethod`. +5. **Warmup**: no core warmup phase; `TextLLMRunner::warmup(prompt, max_new_tokens)` runs a throwaway generation ([text_llm_runner.h#L146](../../executorch/extension/llm/runner/text_llm_runner.h#L146)); Vulkan `optional_warmup_execute`; CUDA-graph warmup steps. +6. Multiple `Method`s per program (e.g., `encoder`, `text_decoder`, `prefill`, `decode`, `vision_encoder`, `token_embedding`, TTS `duration_predictor/text_encoder/vector_estimator/vocoder`) are loaded independently, each with its own planned memory; a global `MergedDataMap` lets them share `.ptd` constants. + +### 4.3 Dynamic shapes [Verified] + +- Export with `torch.export.Dim` dynamic shapes; `ConstraintBasedSymShapeEvalPass` (default in `ExecutorchBackendConfig.sym_shape_eval_pass`) converts symbolic sizes to upper bounds for memory planning (`DynamicMemoryPlanningMode.UPPER_BOUND` default; `SYMBOLIC` exists) ([sym_shape_eval_pass.py#L257](../../executorch/exir/passes/sym_shape_eval_pass.py#L257), [dynamic_shape.py](../../executorch/exir/dynamic_shape.py), [_config.py#L56](../../executorch/exir/capture/_config.py#L56)). +- Runtime tensors carry `TensorShapeDynamism {STATIC, DYNAMIC_BOUND, DYNAMIC_UNBOUND}` ([tensor_shape_dynamism.h](../../executorch/runtime/core/tensor_shape_dynamism.h)); `set_input` calls `resize_tensor` then copies; `DYNAMIC_UNBOUND` is not memory-planned and must be provided by the caller. +- Delegates handle dynamic shapes themselves: XNNPACK `xnn_reshape_external_value` + `xnn_reshape_runtime` + `resize_outputs` per call ([XNNExecutor.cpp#L150](../../executorch/backends/xnnpack/runtime/XNNExecutor.cpp#L150)); Vulkan `maybe_resize_input`/`propagate_resize`; MLX advertises dynamic shape support; QNN graphs are static (wiki known-issues). LLM examples default to static (`enable_dynamic_shape=False` in [model_args.py#L102](../../executorch/examples/models/llama/model_args.py#L102)); the LLM runner reads `enable_dynamic_shape` metadata ([constants.h](../../executorch/extension/llm/runner/constants.h)) and otherwise pads prefill to fixed length. + +### 4.4 State & KV cache [Verified] + +- **In-graph mutable buffers**: `KVCache` registers `k_cache/v_cache` buffers of shape `[max_batch, n_heads, max_context_len, head_dim]` and updates with `index_copy_`/slice writes ([attention.py#L99](../../executorch/examples/models/llama/attention.py#L99)); `CustomKVCache` uses `llama::update_cache` + `llama::custom_sdpa` ([custom_kv_cache.py#L579](../../executorch/examples/models/llama/source_transformation/custom_kv_cache.py#L579), [op_update_cache.cpp](../../executorch/extension/llm/custom_ops/op_update_cache.cpp), [op_sdpa.cpp](../../executorch/extension/llm/custom_ops/op_sdpa.cpp)). Ring buffer (`CustomRingKVCache`, `QuantizedRingKVCache`) and attention-sink variants exist; `ReplaceKVCache...` transforms are selected by `ModelConfig` flags (`use_kv_cache`, `use_sdpa_with_kv_cache`, `quantize_kv_cache`, `use_attention_sink`, `use_ring_kv_cache`). +- The cache lives in the method's planned non-const memory (mutable buffer, never freed) and persists across `execute()` calls; the runner tracks `pos_` and passes `start_pos` as an input tensor. `reset()` zeroes position; there is **no paging, eviction or cross-request sharing** in the core runner — one `Method` = one sequence. `max_context_len` vs `max_seq_len` (prefill chunk) are metadata methods (`get_max_context_len`, `get_max_seq_len`). +- **Static attention (NPU-friendly)**: `StaticKVCache`/`StaticAttention`/`StaticAttentionIOManager` in Python ([static_attention.py](../../executorch/examples/models/llama/static_attention.py)) and C++ ([static_attention_io_manager.h](../../executorch/examples/models/llama/runner/static_attention_io_manager.h)) keep caches *outside* the graph as method inputs/outputs; update styles `SHIFT_POINTER` and `SMART_MASK` (fixed I/O pointers enabling persistent AP↔NPU mapping; supports per-layer cache lengths for sliding-window layers, batch 1, lookahead n-gram cache). Used by the QNN runner ([kv_manager.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/kv_manager.h), [rpc_mem.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/rpc_mem.h)). +- **Off-graph cache library** (new, experimental): `extension/llm/cache` defines `CacheBase`, `SequenceControl::rewind`, `SequencePlanner::plan/commit`, `LayoutPolicy` (`FlatPolicy`, `RingPolicy`), `SequenceCache`, and a paged-like `CellCache` (cells claimed per position with owner bitsets, `kMaxSeqs = 64`, `free_cells()`), reached from the graph through the functional custom op `kvcache::update_and_attend` keyed by `layer_id` ([cache.h](../../executorch/extension/llm/cache/cache.h), [cell_cache.h](../../executorch/extension/llm/cache/cell_cache.h), [sequence_cache.h](../../executorch/extension/llm/cache/sequence_cache.h), [update_and_attend.py](../../executorch/extension/llm/cache/update_and_attend.py)). MLX supports per-session mutable state (`mlx_mutable_state.h`, `skip_mutable_buffer_init` — [MLXBackend.cpp#L298](../../executorch/backends/mlx/runtime/MLXBackend.cpp#L298)); CUDA has `cuda_mutable_state.h` ([backends/cuda/runtime/cuda_mutable_state.h](../../executorch/backends/cuda/runtime/cuda_mutable_state.h)). **[Inferred]** these are the substrate for the multi-session `LLMEngine` implementations (Muse Glimmer, Qwen3.5-MoE, Gemma4-31B engines). + +### 4.5 Multimodal stages [Verified] + +- `MultimodalRunner` (`IRunner`) takes `std::vector` (text / tokens / `Image` / `Audio` (mel) / `RawAudio`); `MultimodalPrefiller` runs `vision_encoder` or `audio_encoder`, `token_embedding`, then `text_decoder` for prefill; decode continues with `MultimodalDecoderRunner`/`TextTokenGenerator` ([multimodal_runner.h](../../executorch/extension/llm/runner/multimodal_runner.h), [multimodal_prefiller.cpp#L112](../../executorch/extension/llm/runner/multimodal_prefiller.cpp#L112), [multimodal_input.h](../../executorch/extension/llm/runner/multimodal_input.h)). +- Encoders/preprocessors as separate methods or `.pte`s: LLaVA (`export_image_encoder`, CLIP preprocessing in Python; [export_llava.py#L138](../../executorch/examples/models/llava/export_llava.py#L138)); Llama 3.2 Vision (exportable preprocess + C++ preprocess in [llama3_2_vision/preprocess](../../executorch/examples/models/llama3_2_vision/preprocess/preprocess.h)); Whisper mel `preprocessor.pte` ([extension/audio/mel_spectrogram.py](../../executorch/extension/audio/mel_spectrogram.py)); Voxtral Realtime `preprocessor.pte` ([voxtral_realtime README](../../executorch/examples/models/voxtral_realtime/README.md)). +- Decoders/vocoders: Voxtral TTS `codec_decoder.pte` (Conv1d/ConvTranspose1d + 8 transformer layers → 24 kHz) driven by `voxtral_tts_runner` with a 7-step Euler flow head; Supertonic `vocoder` method; Mimi decoder on QNN (`qnn_mimi_decoder_runner.cpp`). +- Diffusion: `examples/models/stable_diffusion` and `stable_diffusion_3_5_large` are model definitions/export scripts only (no runner); DiT on QNN ([dit.py](../../executorch/examples/qualcomm/oss_scripts/dit.py)). + +--- + +## 5. Scheduler + +### 5(a) Request / token / pipeline-level scheduling + +- **Core runtime**: **N/A** [Verified]. One `Method` executes one request synchronously; `execute()` refuses re-entry while `in_progress()`; there is no queue, no batching across requests, no admission control. Batch size is baked into the exported shapes (LLM examples: batch 1). +- **LLM runner (`extension/llm/runner`)** [Verified]: sequential prefill → decode per call; `GenerationConfig` (`max_new_tokens`, `seq_len`, `temperature`, `echo`, `warming`, `grammar`) ([irunner.h](../../executorch/extension/llm/runner/irunner.h)); chunked prefill in `TextPrefiller::prefill` when prompt > `max_seq_len` ([text_llm_runner.cpp#L143](../../executorch/extension/llm/runner/text_llm_runner.cpp#L143)); optional two-method PTE (`prefill` + `decode`) ([constants.h#L28](../../executorch/extension/llm/runner/constants.h#L28)). Cancellation: `IRunner::stop()` sets an atomic checked per token in `TextTokenGenerator::generate` ([text_token_generator.h#L131](../../executorch/extension/llm/runner/text_token_generator.h#L131)); `LLMSession::stop()` is documented as a token-boundary cooperative stop ([llm_session.h#L104](../../executorch/extension/llm/runner/llm_session.h#L104)). No backpressure concept (callbacks are synchronous on the calling thread). +- **`extension/llm/batching` (experimental)** [Verified]: `Scheduler` interface (`submit/has_work/get_work/cancel(sid)/clear/max_prefill_chunk_size`), `DecodeFirstScheduler::create(max_batch_tokens=544, max_decode_sequences=32, max_prefill_chunk_size=256)` takes all pending decodes first then fills the token budget with round-robin prefill chunks (continuous batching with chunked prefill; admission rejects over-budget tasks) ([decode_first_scheduler.h](../../executorch/extension/llm/batching/decode_first_scheduler.h)); `Executor` (`open_session/close_session/set_sampling/execute(BatchInput, BatchOutput)`); `Runner(executor, scheduler)` with `open_session_async`, `Session::generate_async(delta, GenConfig, callback)` returning `GenerationHandle{cancel, done, wait, finish_reason}` and `FinishReason {StopToken, NewTokenLimit, Cancelled, Failed}` ([runner.h](../../executorch/extension/llm/batching/runner.h)). **Gap [Verified]**: no in-tree `Executor` implementation outside `extension/llm/batching/test`; no batched `.pte` example wires into it yet. +- **`examples/llm_server`** [Verified]: OpenAI-compatible `/v1/chat/completions` (streaming + non-streaming), one worker process with *serialized execution*, multiple isolated sessions on one weight load when the engine reports `LLMServingCapacity > 1`, bounded request cancellation, structured 400s for unsupported params (`n>1`, penalties, logprobs...) ([examples/llm_server/README.md](../../executorch/examples/llm_server/README.md), [worker_loop.h](../../executorch/examples/llm_server/cpp/worker_loop.h), [worker_prefill_plan.h](../../executorch/examples/llm_server/cpp/worker_prefill_plan.h)). `LLMEngine` implementations: [muse_glimmer_engine.h](../../executorch/examples/models/muse-glimmer/runtime/engine/muse_glimmer_engine.h) (with DFlash speculative decoding), [qwen35_moe_engine.h](../../executorch/examples/models/qwen3_5_moe/qwen35_moe_engine.h), [gemma4_31b_engine.h](../../executorch/examples/models/gemma4_31b/gemma4_31b_engine.h). +- **Multi-stage pipelines** (TTS/ASR): hand-written in each example runner (e.g., `VoxtralTtsRunner::synthesize_streaming` LM → flow head → codec chunks with `streaming_chunk_frames_=25`, `streaming_initial_chunk_=5`, `streaming_left_context_=25` — [voxtral_tts_runner.h#L121](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.h#L121)); QNN has a Python `genai_pipeline` with model-preparation / quantization / compilation / inference stages ([genai_pipeline.py](../../executorch/backends/qualcomm/genai_pipeline/genai_pipeline.py)). + +### 5(b) Graph / operator / thread-level scheduling + +- **Op ordering** [Verified]: fixed at export (topological order emitted into `Chain.instructions`); executed strictly sequentially by `Method::execute` — "Chains are executed sequentially today, but future async designs may branch and run many in parallel" ([method.cpp#L1722](../../executorch/runtime/executor/method.cpp#L1722)). No inter-op parallelism, no stream assignment in the core. +- **Intra-op threading** [Verified]: `extension/threadpool` wraps `pthreadpool` and sizes the pool from `cpuinfo` performant cores (`get_num_performant_cores`, [cpuinfo_utils.cpp](../../executorch/extension/threadpool/cpuinfo_utils.cpp)); `ThreadPool::run(fn, range)` is blocking `pthreadpool_parallelize_1d`; `_unsafe_reset_threadpool(n)` changes size ([threadpool.cpp#L65](../../executorch/extension/threadpool/threadpool.cpp#L65)); kernels use `parallel_for(begin,end,grain,fn)` which degrades to a serial loop without `ET_USE_THREADPOOL` ([thread_parallel_interface.h#L53](../../executorch/runtime/kernel/thread_parallel_interface.h#L53)). XNNPACK shares the same `get_pthreadpool()` ([XNNCompiler.cpp#L2300](../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp#L2300)). `NoThreadPoolGuard` prevents nested parallelism. +- **Delegate-internal scheduling** [Verified]: Vulkan builds command buffers per graph and submits/waits (`submit_current_cmd_and_wait`); CUDA optionally replays a captured CUDA graph; QNN executes a named graph on HTP with optional multi-graph context; XNNPACK runs the subgraph on the pthreadpool. +- **Batching** [Verified]: static batch dim only; `HierarchicalAllocator` sizes are per-method so batch changes require re-export. + +--- + +## 6. I/O layer + +- **Text tokenization** [Verified via docs, submodule absent]: `extension/llm/tokenizers` (pytorch-labs tokenizers, submodule commit `1d7ca636`) provides C++ `HFTokenizer` (tokenizer.json), `Tiktoken`, `SentencePiece`, `Llama2c`, `Tekken` plus Python `pytorch_tokenizers.get_tokenizer` ([.claude/tokenizers.md](../../executorch/.claude/tokenizers.md)); `llm::load_tokenizer(path, special_tokens)` auto-detects ([llm_runner_helper.h#L47](../../executorch/extension/llm/runner/llm_runner_helper.h#L47)); llama runner tries Tiktoken → SentencePiece → Llama2c ([runner.cpp](../../executorch/examples/models/llama/runner/runner.cpp)). WASM tokenizers bindings in [extension/wasm/tokenizers](../../executorch/extension/wasm/tokenizers). +- **Image preprocessing** [Verified]: C++ `ImageProcessor::process/process_yuv/process_into` (resize + normalize, NV12/NV21, SIMD and Apple GPU variants) in [extension/image/image_processor.h](../../executorch/extension/image/image_processor.h); Swift `ExecuTorch+ImageProcessor.swift`; JNI `jni_layer_image.cpp`; examples use `stb_image` ([gemma3/e2e_runner.cpp](../../executorch/examples/models/gemma3/e2e_runner.cpp)) and `llm::Image` ([image.h](../../executorch/extension/llm/runner/image.h)); custom op `tile_crop` ([op_tile_crop.cpp](../../executorch/extension/llm/custom_ops/op_tile_crop.cpp)). +- **Audio** [Verified]: `WhisperAudioProcessor` (torch.stft + mel filterbank, exportable to `whisper_preprocess.pte`, 16 kHz/80 bins/30 s defaults) in [extension/audio/mel_spectrogram.py](../../executorch/extension/audio/mel_spectrogram.py); `RawAudio{data, batch, n_channels, n_samples}` and `Audio` (mel, float/bf16) types ([audio.h](../../executorch/extension/llm/runner/audio.h)); `wav_loader.h`; ASR runners `Seq2SeqRunner` (Whisper-style `encoder` + `text_decoder`) and `TransducerRunner` (Parakeet TDT) in [extension/asr/runner](../../executorch/extension/asr/runner/seq2seq_runner.h); Silero VAD streaming runner ([stream_main.cpp](../../executorch/examples/models/silero_vad/stream_main.cpp)). +- **Streaming** [Verified]: token-level `token_callback(const std::string&)` per generated token and `stats_callback(const Stats&)` (`first_token_ms`, `prompt_eval_end_ms`, `num_generated_tokens`, `gpu_total_bytes` — [stats.h](../../executorch/extension/llm/runner/stats.h)); audio-chunk `AudioChunkCallback` in Voxtral TTS; Voxtral Realtime processes 80 ms chunks live; Supertonic persistent mode emits JSONL results per request. HTTP SSE streaming in `examples/llm_server`. +- **Buffering** [Verified]: planned arenas allocated by the app; `Module` allocates them internally; `set_output_data_ptr` for zero-copy outputs; XNNPACK weights cache for `.ptd` weights ([XNNCompiler.cpp#L2283](../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp#L2283)). +- **Postprocessing** [Verified]: `Sampler` (temperature, top-k `sample_topk`, argmax) + `LogitProcessor` chain (grammar hooks) ([sampler.h](../../executorch/extension/llm/sampler/sampler.h), [logit_processor.h](../../executorch/extension/llm/sampler/logit_processor.h)); tokenizer `decode(prev, cur)`; WAV writers in TTS examples ([wav_writer.h](../../executorch/examples/models/voxtral_tts/wav_writer.h)). +- **Application boundaries** [Verified]: + - C++: `runtime/` core API, `extension/module::Module`, `extension/tensor` (`TensorPtr`), `extension/runner_util`, LLM/ASR runners, `executor_runner` CLI ([examples/portable/executor_runner/executor_runner.cpp](../../executorch/examples/portable/executor_runner/executor_runner.cpp)). + - Python: `executorch.runtime.Runtime/Program/Method` and `portable_lib._load_for_executorch` ([.claude/runtime-api.md](../../executorch/.claude/runtime-api.md), [extension/pybindings/pybindings.cpp](../../executorch/extension/pybindings/pybindings.cpp)); LLM runner pybindings ([extension/llm/runner/pybindings.cpp](../../executorch/extension/llm/runner/pybindings.cpp)). + - Android: Kotlin `Module`, `Tensor`, `EValue`, `ExecuTorchRuntime`, `BackendOptionsMap` + JNI layers for runtime/llama/asr/image/training ([extension/android](../../executorch/extension/android/README.md), [jni_layer_llama.cpp](../../executorch/extension/android/jni/jni_layer_llama.cpp)); shipped as AAR on Maven Central ([using-executorch-android.md](../../executorch/docs/source/using-executorch-android.md)). + - Apple: Objective-C/Swift `ExecuTorchModule/Tensor/Value/ImageProcessor/BackendOptions` ([extension/apple/ExecuTorch/Exported](../../executorch/extension/apple/ExecuTorch/Exported/ExecuTorchModule.h)) and `ExecuTorchLLM` ([extension/llm/apple](../../executorch/extension/llm/apple)); xcframeworks via SwiftPM ([using-executorch-ios.md](../../executorch/docs/source/using-executorch-ios.md)). + - WASM: [extension/wasm/wasm_bindings.cpp](../../executorch/extension/wasm/wasm_bindings.cpp). + - HTTP: `examples/llm_server` (Python control plane; C++ single-binary server listed as future). + - Demo apps in tree: only [examples/demo-apps/react-native/rnllama](../../executorch/examples/demo-apps/react-native) (native Android/iOS demo apps have moved out of this tree; docs still reference them). + +--- + +## 7. Execution flow + +### 7(a) Text LLM decode (TextLLMRunner over XNNPACK-delegated `.pte`) [Verified] + +```mermaid +sequenceDiagram + participant App + participant R as TextLLMRunner + participant Tok as Tokenizer + participant P as TextPrefiller + participant G as TextTokenGenerator + participant D as TextDecoderRunner + participant M as Module/Method + participant K as Kernels / XNNPACK delegate + participant S as Sampler + App->>R: create_text_llm_runner(pte, tokenizer, data_files) + App->>R: load() + R->>M: load_method("forward") → Method::init (parse_values, BackendDelegate::Init → XNNPACKBackend::init builds xnn runtime, resolve_operator) + App->>R: generate(prompt, GenerationConfig, token_cb, stats_cb) + R->>Tok: encode(prompt, bos, eos) + R->>P: prefill(tokens, pos_) (chunked ≤ max_seq_len) + P->>D: step(tokens_tensor, start_pos) + D->>M: IOManager::prepare_prefill → execute("forward", {tokens, start_pos}) + M->>K: Method::execute: KernelCall/DelegateCall sequence (update_cache, custom_sdpa, xnn subgraph) + K-->>M: logits + M-->>D: logits tensor + D->>S: logits_to_token(temperature) + S-->>P: first token + P-->>R: first token, pos_ += n + R->>G: generate(first token, start_pos, max_new_tokens, temperature, token_cb) + loop until EOS / max_new_tokens / stop() + G->>D: step({cur_token}, pos) + D->>M: execute("forward"|"decode") + M->>K: instructions (KV cache mutable buffer updated in planned memory) + K-->>D: logits + D->>S: sample → cur_token + G->>Tok: decode(prev, cur) + G-->>App: token_cb(text) + end + G-->>R: n_generated + R-->>App: stats_cb(Stats: TTFT, tok/s ...) +``` + +Source: [text_llm_runner.cpp](../../executorch/extension/llm/runner/text_llm_runner.cpp), [text_prefiller.h](../../executorch/extension/llm/runner/text_prefiller.h), [text_token_generator.h#L83](../../executorch/extension/llm/runner/text_token_generator.h#L83), [text_decoder_runner.cpp#L40](../../executorch/extension/llm/runner/text_decoder_runner.cpp#L40), [io_manager.h](../../executorch/extension/llm/runner/io_manager/io_manager.h), [method.cpp#L1697](../../executorch/runtime/executor/method.cpp#L1697). + +### 7(b) TTS: Voxtral TTS streaming synthesis (CPU or CUDA) [Verified structure; per-method details Inferred] + +```mermaid +sequenceDiagram + participant App + participant TR as VoxtralTtsRunner + participant Tok as Tekken tokenizer + participant LM as model.pte (5 methods: LM + flow head) + participant CD as codec_decoder.pte + participant BE as XNNPACK / CUDA(AOTI) delegate + App->>TR: ctor(model_path, codec_path, tokenizer, voice embedding, [aoti .ptd]) + TR->>LM: Module::load_method(...) for LM + flow-head methods + TR->>CD: Module::load_method("forward") + App->>TR: synthesize_streaming(text, AudioChunkCallback) + TR->>Tok: encode(text) + TR->>LM: prefill(text tokens + voice embedding) + loop per audio frame (autoregressive) + TR->>LM: decode step → hidden state + TR->>LM: flow head: 7-step Euler ODE (x0 from xorshift64 rng) → 37 codebook tokens + LM->>BE: DelegateCall (AOTI container run / xnn runtime) + end + Note over TR: after streaming_initial_chunk_ (5 frames) then every streaming_chunk_frames_ (25) with left context 25 + TR->>CD: execute(codebook tokens window) → PCM 24 kHz + CD->>BE: DelegateCall + TR-->>App: AudioChunkCallback(float PCM chunk) + TR-->>App: final WAV via wav_writer +``` + +Source: [voxtral_tts_runner.h](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.h), [voxtral_tts_runner.cpp](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.cpp), [README](../../executorch/examples/models/voxtral_tts/README.md). Exact method names inside `model.pte` were not enumerated here [Unknown]. + +Multimodal VLM path (for VLA-style vision+text) is the `MultimodalRunner` sequence: `vision_encoder(image)` → `token_embedding(tokens)` → `text_decoder(embeds, start_pos)` prefill → token loop ([multimodal_prefiller.cpp](../../executorch/extension/llm/runner/multimodal_prefiller.cpp)). + +--- + +## 8. Tests and examples + +- **Python tests** [Verified]: `pytest -n auto` from repo root ([CLAUDE.md](../../executorch/CLAUDE.md)); [pytest.ini](../../executorch/pytest.ini) ignores `backends/arm/**` (needs `examples/arm/setup.sh`), some XNNPACK op tests, `backends/test` (WIP harness), LLaVA tests, backend-specific Muse Glimmer pipelines. Tests live next to code: `exir/tests`, `exir/backend/test`, `backends/*/test`, `extension/llm/export/test`, `examples/models/llama/tests` (incl. `test_static_attention.py`), `runtime/test/test_runtime.py`. +- **C++ tests** [Verified]: gtest via CMake `EXECUTORCH_BUILD_TESTS=ON` then `ctest --output-on-failure`; [Test.cmake](../../executorch/Test.cmake) enumerates `runtime/*/test`, `kernels/test`, `extension/*/test`; [test/run_oss_cpp_tests.sh](../../executorch/test/run_oss_cpp_tests.sh); size tests [test/build_size_test.sh](../../executorch/test/build_size_test.sh) and [test/size_test.cpp](../../executorch/test/size_test.cpp); end-to-end [test/end2end](../../executorch/test/end2end). +- **Backend op-coverage harness**: [backends/test](../../executorch/backends/test/README.md) (`suite`, `harness`, FACTO), `multi_method_delegate_test.cpp`. +- **CI** [Verified]: ~60 workflows in [.github/workflows](../../executorch/.github/workflows) — `pull.yml`, `trunk`/`periodic.yml`, `nightly.yml`, `_unittest.yml`, `_android.yml`, `apple.yml`, `cuda.yml`, `cuda-windows.yml`, `cuda-perf.yml`, `rocm.yml`, `metal.yml`, `mlx.yml`, `riscv64.yml`, `qnn-windows-msvc.yml`, `test-backend-{arm,coreml,cortex-m,nxp,openvino}.yml`, `_test_cadence.yml`, `_xtensa_*.yml`, `_llm_server.yml`, `_test_arduino_library.yml`, wheel builds for linux/macos/windows/aarch64/cuda. +- **Example apps / runners** [Verified]: LLM: [examples/models/llama](../../executorch/examples/models/llama/README.md) (`main.cpp`, `runner/`, `export_llama.py`, eval, Android perf table), `examples/models/{qwen3,phi-3-mini,gemma3,gemma4,smollm2,lfm2,deepseek-r1-distill-llama-8B,...}`; VLM: llava, llama3_2_vision, gemma3, smolvlm, internvl3, muse_glimmer; audio: whisper, voxtral, voxtral_realtime, parakeet, silero_vad, sortformer, granite_speech, emformer_rnnt, wav2letter; TTS: voxtral_tts, supertonic; vision: mobilenet_v2/v3, resnet, deit, dinov2, efficient_sam, yolo12/26, torchvision_vit, edsr; diffusion: stable_diffusion, stable_diffusion_3_5_large. Platform examples: [examples/qualcomm](../../executorch/examples/qualcomm/README.md) (40+ oss_scripts incl. llama, whisper, t5, gemma4, moshi/mimi, DiT, EfficientSAM), [examples/apple/coreml](../../executorch/examples/apple/coreml/README.md), [examples/arm](../../executorch/examples/arm/README.md) (Ethos-U/VGF/Cortex-M notebooks: smollm2, silero_vad, tinystories, mobilesam), [examples/mediatek](../../executorch/examples/mediatek/README.md), [examples/samsung](../../executorch/examples/samsung), [examples/nxp](../../executorch/examples/nxp), [examples/openvino](../../executorch/examples/openvino), [examples/vulkan](../../executorch/examples/vulkan), [examples/xnnpack](../../executorch/examples/xnnpack/README.md), [examples/cuda](../../executorch/examples/cuda/README.md) (ROCm pointwise), [examples/arduino](../../executorch/examples/arduino/README.md), [examples/espressif](../../executorch/examples/espressif/README.md), [examples/raspberry_pi/pico2](../../executorch/examples/raspberry_pi/pico2), [examples/riscv](../../executorch/examples/riscv/README.md), [examples/zephyr](../../executorch/examples/zephyr), [examples/wasm](../../executorch/examples/wasm), [examples/llm_pte_finetuning](../../executorch/examples/llm_pte_finetuning), [examples/demo-apps/react-native](../../executorch/examples/demo-apps/react-native). +- **Make targets** [Verified]: [Makefile](../../executorch/Makefile) + per-example `CMakePresets.json` (`make whisper-cpu|whisper-cuda|whisper-metal`, `make silero-vad-cpu`, `make sortformer-cpu`, `make supertonic-mlx`). + +--- + +## 9. TTS relevance + +**Exists** [Verified]: +- **Voxtral-4B-TTS** end-to-end: export ([export_voxtral_tts.py](../../executorch/examples/models/voxtral_tts/export_voxtral_tts.py)) producing `model.pte` (Mistral 4B decoder + 3-layer flow-matching head, 5 methods) and `codec_decoder.pte`; backends `portable`, `xnnpack`, `cuda`, `cuda-windows`; quant `--qlinear 4w|8w|8da4w|8da8w`, `--qlinear-codec`, `--qembedding`; `--streaming` chunk metadata; C++ runner with `synthesize_streaming` + `AudioChunkCallback`, xorshift RNG for flow noise, WAV writer. Repo-reported (not reproduced): CUDA 4w RTF 0.31× on RTX 5080 with ~2.6 s time-to-first-audio, 3.4 GB `.ptd`; FP32 CUDA RTF 51× (codec on portable CPU) ([README](../../executorch/examples/models/voxtral_tts/README.md)). +- **Supertonic 3 on MLX**: single dynamic FP16 `.pte` with methods `duration_predictor`, `text_encoder`, `vector_estimator` (5 flow steps), `vocoder`; one-shot and persistent JSONL server mode; repo-reported RTF ≈ 0.028 on Apple Silicon ([README](../../executorch/examples/models/supertonic/README.md), [supertonic_runner.cpp](../../executorch/examples/models/supertonic/runtime/supertonic_runner.cpp), [model/vocoder.py](../../executorch/examples/models/supertonic/model/vocoder.py)). macOS/Apple-Silicon only. +- **Neural audio codecs**: Mimi (Moshi) tests ([examples/models/moshi/mimi/test_mimi.py](../../executorch/examples/models/moshi/mimi/test_mimi.py)) and a QNN Mimi decoder runner ([qnn_mimi_decoder_runner.cpp](../../executorch/examples/qualcomm/oss_scripts/moshi/qnn_mimi_decoder_runner.cpp)); Voxtral codec decoder. +- **Audio ops**: portable/optimized FFT (`op_fft_r2c/c2r` in [kernels/optimized/cpu](../../executorch/kernels/optimized/cpu/op_fft_r2c.cpp), pocketfft third-party), Conv1d→Conv2d transform ([backends/transforms/convert_conv1d_to_conv2d_pass.py](../../executorch/backends/transforms/convert_conv1d_to_conv2d_pass.py)) for XNNPACK; `ConvTranspose1d` handled in Voxtral codec on CPU/CUDA; mel front end exportable. +- **Streaming audio output**: chunked codec decoding with left context (Voxtral TTS); real-time 80 ms chunk ASR (Voxtral Realtime) shows the runtime can sustain streaming loops; VAD streaming runner. +- **Text front end**: Supertonic `text_processor.cpp`; no general G2P/phonemizer library. +- **NXP** README explicitly lists TTS as a Neutron NPU use-case ([backends/nxp/README.md](../../executorch/backends/nxp/README.md)) [Verified statement, no TTS example for NXP found]. + +**Missing / gaps** [Verified unless noted]: +- No generic TTS runner abstraction (unlike `extension/llm/runner` or `extension/asr/runner`); each TTS model has a bespoke C++ runner. +- No mobile (Android/iOS/QNN/CoreML) TTS example; Supertonic is MLX-only, Voxtral TTS is CPU/CUDA. [Inferred] the XNNPACK path is fp32 (`--dtype fp32` default) and 4B params — heavy for phones. +- No vocoder zoo (HiFi-GAN, BigVGAN, WaveRNN, etc.) or codec encoders (EnCodec/SNAC/DAC) in tree; `stable_diffusion` has no audio variant. +- Flow-matching ODE loops are unrolled by the runner (7 Euler steps as repeated method calls), not expressed as a graph; no CFG batching helper. +- No audio-specific streaming buffer/jitter management; callbacks are synchronous. +- Sampling for audio tokens (multi-codebook, 37 codebooks per frame) is handled inside the example, not by `Sampler`. + +--- + +## 10. VLA relevance + +**Exists** [Verified]: +- **Vision encoders**: CLIP (LLaVA), Llama 3.2 Vision encoder + cross-attention decoder, SigLIP-style Gemma3/SmolVLM/InternVL3 via optimum-executorch converters ([examples/models/smolvlm/convert_weights.py](../../executorch/examples/models/smolvlm/convert_weights.py)), DINOv2, ViT, DeiT, EfficientSAM, YOLO, MobileNet; `MultimodalRunner` handles image → embeddings → decoder; C++ `ImageProcessor` for resize/normalize/YUV. +- **Structured inputs**: methods accept arbitrary tensor tuples (`Method::set_input`), so proprioceptive vectors are trivially additional inputs [Inferred: no example does this]. +- **Low-latency loops**: static memory planning, no allocation in steady state, `set_output_data_ptr` zero-copy outputs, CUDA-graph replay for fixed-shape methods, Vulkan/QNN prepacked graphs, `Method::execute` overhead is a flat instruction walk — good fit for fixed-rate control loops. Static attention IO manager demonstrates fixed-pointer I/O for NPUs. +- **Diffusion/flow machinery**: SD/SD3.5 model definitions, DiT on QNN, Voxtral/Supertonic flow-matching heads driven by runner-side Euler loops; `MoveCall`/`JumpFalseCall` allow `torch.cond`/`while` control flow in a single method (`get_while_nodes`, `get_scan_nodes` in memory planning) — [Inferred] a fixed-step denoising loop could be exported as `torch.ops.higher_order.while_loop` but no example does so. +- **Action heads**: nothing VLA-specific; an MLP/transformer action head is just another method. + +**Missing** [Verified]: +- No VLA models (OpenVLA, π0, ACT, Diffusion Policy, SmolVLA, Octo), no LeRobot integration, no "robot" mentions in docs/examples/wiki. +- No multi-method orchestration primitive for encoder + policy + action-decoder pipelines beyond writing a C++ runner; no shared-memory/DMA-BUF camera input path (image bytes must be copied into an ExecuTorch tensor). +- No batched multi-camera encoder execution beyond exporting a batched shape. +- Jetson/embedded-CUDA deployment is undocumented (see §11). + +**[Proposal]** For an edge VLA on ExecuTorch: export `vision_encoder` (per camera, batched), `proprio_proj`, `text/token_embedding`, `policy_decoder` (static KV, `SMART_MASK` style) and `action_head` (flow/diffusion step as a method called N times) into one `.pte` with shared `.ptd` weights; drive with a small C++ runner modelled on `MultimodalPrefiller` + `VoxtralTtsRunner`; pin planned arenas once and reuse `Method` objects across control ticks. + +--- + +## 11. Edge deployment profile + +- **Binary size** [Verified, repo-reported]: docs state the core library is "around 50 kB with no operators/kernels or delegates" ([kernel-library-selective-build.md](../../executorch/docs/source/kernel-library-selective-build.md)); CI thresholds in [pull.yml](../../executorch/.github/workflows/pull.yml#L555): stripped `size_test` ≤ 52,168 B (gcc9) / ≤ 45,000 B (clang); bare-metal Arm `size_test` ≤ 118,000 B (GCC 15.2 baseline 115,868 B) and Zephyr preset ≤ 136,020 B. Size tooling: `EXECUTORCH_OPTIMIZE_SIZE` (`-Os -fno-exceptions -fno-rtti`), `ET_LOG_ENABLED=0`, bloaty ([.claude/skills/binary-size/SKILL.md](../../executorch/.claude/skills/binary-size/SKILL.md), [devtools/size_analysis_tool](../../executorch/devtools/size_analysis_tool)). +- **Selective build** [Verified]: `EXECUTORCH_SELECT_OPS_YAML|LIST|MODEL`, `EXECUTORCH_ENABLE_DTYPE_SELECTIVE_BUILD`, `gen_selected_ops(...)` macro; generates `selected_max_kernel_num.h` ([tools/cmake/Codegen.cmake](../../executorch/tools/cmake/Codegen.cmake), [examples/selective_build](../../executorch/examples/selective_build)). +- **Dependencies** [Verified]: core needs only C++17 and flatbuffers headers (`third-party/flatbuffers`); optional pthreadpool + cpuinfo (`EXECUTORCH_BUILD_PTHREADPOOL`, `EXECUTORCH_BUILD_CPUINFO`), XNNPACK, Eigen (optimized BLAS), gflags for runners, flatcc for ETDump, torchao (`third-party/ao`) for quant kernels, tokenizers submodule for LLM; C++ `c10` headers vendored (`check-c10-sync.yml`). Python side pins PyTorch via [torch_pin.py](../../executorch/torch_pin.py). +- **Build system** [Verified]: CMake (`CMakeLists.txt`, presets in [tools/cmake/preset](../../executorch/tools/cmake/preset/default.cmake) with `EXECUTORCH_BUILD_{XNNPACK,VULKAN,QNN,COREML,METAL,MLX,CUDA,ROCM,NEURON,ENN,NXP_NEUTRON,OPENVINO,CADENCE,CORTEX_M,VGF,ARM_BAREMETAL,ARM_ETHOSU_LINUX,WEBGPU,WASM,...}`, `EXECUTORCH_BUILD_EXTENSION_{MODULE,TENSOR,DATA_LOADER,FLAT_TENSOR,LLM,LLM_RUNNER,ASR_RUNNER,IMAGE,TRAINING,APPLE,...}`, `EXECUTORCH_BUILD_KERNELS_{OPTIMIZED,QUANTIZED,LLM}`); Buck2 (`BUCK`/`targets.bzl`) for Meta-internal; Gradle for AAR; `Package.swift` for SwiftPM; Arduino library builder; `pip install executorch` wheels for Linux/macOS/Windows (+ CUDA and aarch64-CUDA wheels). +- **Platform matrix** [Verified from backend docs/dirs]: + - **Embedded NVIDIA / Jetson**: [Unknown] — CUDA backend documented only for "Linux or Windows" x86-class hosts; aarch64 Linux CUDA wheels are built in CI, which [Inferred] would cover Jetson/GB10-class aarch64 if the AOTI `.so` is compiled for the target SM, but no doc, example or CI run targets Jetson. + - **ARM CPU**: XNNPACK (ARM64/ARMv7, KleidiAI i8mm/dotprod kernels by default on ARM), torchao low-bit kernels (`torchao:8da4w`), `riscv64-linux` preset also exists; Raspberry Pi tutorial ([raspberry_pi_llama_tutorial.md](../../executorch/docs/source/raspberry_pi_llama_tutorial.md)). + - **Cortex-M / MCU**: `arm-baremetal`, `zephyr`, `esp-baremetal` presets; Cortex-M CMSIS-NN dialect; Ethos-U55/65/85 delegate; Arduino library; Pico2 example; Alif/Zephyr tutorial ([zephyr_alif_tutorial.md](../../executorch/docs/source/zephyr_alif_tutorial.md)). + - **Apple**: CoreML (ANE/GPU/CPU, iOS/macOS), MLX (Apple-Silicon GPU, macOS ≥ 14, iOS ≥ 17 experimental), Metal (experimental AOTI), XNNPACK; distributed as xcframeworks `executorch, executorch_llm, backend_coreml, backend_mlx, backend_xnnpack, kernels_llm/optimized/quantized/torchao`. + - **Android**: XNNPACK, Vulkan (Adreno/Mali; Maven artifact `executorch-android-vulkan`), QNN HTP (Snapdragon; 38 chipset ids incl. automotive SA8295/SA8797), MediaTek Neuron (D9300/D9400), Samsung ENN (Exynos 2500), Arm VGF (Vulkan ML). NNAPI: not present in tree. OpenCL: not present (Vulkan only). + - **Intel**: OpenVINO CPU/iGPU/dGPU/NPU. + - **Browser**: WebGPU + WASM bindings. +- **Model conversion constraints** [Verified]: must be `torch.export`-able (no data-dependent Python control flow except `torch.cond`/`while_loop`; static or bounded dynamic shapes); one `.pte` per backend; QNN/Ethos-U/CoreML prefer static shapes and quantized graphs; Ethos-U requires Vela-compilable int8 TOSA; dim-order/channels-last pitfalls documented in [.wiki/export/common-pitfalls.md](../../executorch/.wiki/export/common-pitfalls.md); CUDA export needs `EdgeCompileConfig(_check_ir_validity=False, _skip_dim_order=True)` ([cuda-overview.md](../../executorch/docs/source/backends/cuda/cuda-overview.md)); PTE BC guarantee is ≥ one following minor release ([runtime/COMPATIBILITY.md](../../executorch/runtime/COMPATIBILITY.md)). +- **Repo-reported perf (not reproduced)**: Llama 3.2 1B SpinQuant on OnePlus 12 (XNNPACK+KleidiAI): 50.2 tok/s decode, 0.3 s TTFT, 1,083 MiB PTE, 1,921 MiB RSS; 3B SpinQuant 19.7 tok/s ([llama/README.md#L70](../../executorch/examples/models/llama/README.md#L70)); WebGPU Llama 3.2 1B 4-bit on M4 Pro 188 tok/s decode @128 ctx ([webgpu README](../../executorch/backends/webgpu/README.md)). + +--- + +## 12. Limitations and unknowns + +- [Verified] Single-method, synchronous, sequential execution; no request-level batching in the core; `extension/llm/batching` has no production `Executor` implementation in tree. +- [Verified] Global static registries (backends ≤ 16, kernels ≤ 2000 by default) with an explicit TODO that multiple `Executor` instances are not supported in the long term. +- [Verified] Backend availability is checked only at delegate init; no runtime fallback if a delegate is unavailable — a `.pte` is tied to its backend. +- [Verified] KV cache is a static, pre-sized mutable buffer per method; no paging/eviction/prefix sharing in the core (experimental `CellCache`/`SequenceCache` exist but are wired only through custom ops and engine examples). +- [Verified] Dynamic shapes are upper-bound planned; `DYNAMIC_UNBOUND` outputs need caller-provided memory; several NPU delegates (QNN, Ethos-U) are static-shape only per wiki. +- [Verified] No VLA/policy models, no robotics integration, no general TTS runner, no vocoder/codec library; TTS examples are CUDA/MLX-centric. +- [Verified] Tokenizers, XNNPACK, pthreadpool, cpuinfo, MLX, Vulkan headers, flatbuffers, torchao are uninitialized submodules in this checkout — their internals (e.g., tokenizer implementations, XNNPACK version, KleidiAI integration details) could not be inspected. +- [Unknown] Jetson/embedded CUDA support status; whether AOTI-compiled `.so` blobs are portable across SM versions and aarch64. +- [Unknown] Exact method set and shapes inside Voxtral TTS `model.pte`; exact QNN LLM multi-graph memory layout; Metal and Core AI backend maturity. +- [Unknown] `examples/llm_server` C++ single-binary server ("future"); Android/iOS native demo apps are referenced by docs but not present in tree (only react-native). +- [Inferred] Cross-method sharing of a KV cache (e.g., encoder→decoder cross-attn cache, `update_cross_attn_cache` op) is supported by custom ops but memory planning treats each method independently, so shared state must be threaded as inputs/outputs or held off-graph. +- [Inferred] Thread-count control is global (`_unsafe_reset_threadpool`), so co-scheduling two models (e.g., VLM + TTS) on CPU has no per-model thread budget. + +--- + +## 13. Reference index + +| Path | Role | +|---|---| +| [../../executorch/CLAUDE.md](../../executorch/CLAUDE.md) | Repo agent guidance, quick build/test commands | +| [../../executorch/.claude/backends.md](../../executorch/.claude/backends.md) | Backend table + partitioner imports | +| [../../executorch/.claude/runtime-api.md](../../executorch/.claude/runtime-api.md) | Python runtime API summary | +| [../../executorch/.claude/quantization.md](../../executorch/.claude/quantization.md) | Quantizer list | +| [../../executorch/.claude/llm-export.md](../../executorch/.claude/llm-export.md) | LLM export config summary | +| [../../executorch/.claude/tokenizers.md](../../executorch/.claude/tokenizers.md) | Tokenizer classes (submodule absent) | +| [../../executorch/.claude/faq.md](../../executorch/.claude/faq.md) | Error codes, threadpool reset | +| [../../executorch/.claude/skills/binary-size/SKILL.md](../../executorch/.claude/skills/binary-size/SKILL.md) | Size measurement procedure | +| [../../executorch/.wiki/index.md](../../executorch/.wiki/index.md) | Tribal knowledge index | +| [../../executorch/.wiki/export/model-specific.md](../../executorch/.wiki/export/model-specific.md) | LLM export codepaths, dynamic-shape limits | +| [../../executorch/.wiki/export/common-pitfalls.md](../../executorch/.wiki/export/common-pitfalls.md) | Export pitfalls | +| [../../executorch/.wiki/quantization/recipes.md](../../executorch/.wiki/quantization/recipes.md) | Quant recipe selection | +| [../../executorch/version.txt](../../executorch/version.txt) | Version 1.5.0 | +| [../../executorch/CMakePresets.json](../../executorch/CMakePresets.json) | Build presets | +| [../../executorch/tools/cmake/preset/default.cmake](../../executorch/tools/cmake/preset/default.cmake) | `EXECUTORCH_BUILD_*` options | +| [../../executorch/tools/cmake/Codegen.cmake](../../executorch/tools/cmake/Codegen.cmake) | Selective build macros | +| [../../executorch/codegen/gen.py](../../executorch/codegen/gen.py) | Kernel registration codegen | +| [../../executorch/Makefile](../../executorch/Makefile) | Runner make targets | +| [../../executorch/Test.cmake](../../executorch/Test.cmake) | C++ test enumeration | +| [../../executorch/pytest.ini](../../executorch/pytest.ini) | Python test config | +| [../../executorch/test/build_size_test.sh](../../executorch/test/build_size_test.sh) | Size test build | +| [../../executorch/test/size_test.cpp](../../executorch/test/size_test.cpp) | Minimal runtime binary | +| [../../executorch/.github/workflows/pull.yml](../../executorch/.github/workflows/pull.yml) | CI incl. size thresholds | +| [../../executorch/.github/workflows/build-wheels-cuda-aarch64-linux.yml](../../executorch/.github/workflows/build-wheels-cuda-aarch64-linux.yml) | aarch64 CUDA wheels | +| [../../executorch/schema/program.fbs](../../executorch/schema/program.fbs) | PTE schema | +| [../../executorch/schema/extended_header.h](../../executorch/schema/extended_header.h) | PTE extended header | +| [../../executorch/runtime/COMPATIBILITY.md](../../executorch/runtime/COMPATIBILITY.md) | PTE/runtime BC policy | +| [../../executorch/runtime/executor/program.h](../../executorch/runtime/executor/program.h) | `Program` API | +| [../../executorch/runtime/executor/method.h](../../executorch/runtime/executor/method.h) | `Method` API | +| [../../executorch/runtime/executor/method.cpp](../../executorch/runtime/executor/method.cpp) | `BackendDelegate`, init/execute/step | +| [../../executorch/runtime/executor/memory_manager.h](../../executorch/runtime/executor/memory_manager.h) | `MemoryManager` | +| [../../executorch/runtime/core/hierarchical_allocator.h](../../executorch/runtime/core/hierarchical_allocator.h) | Planned arenas | +| [../../executorch/runtime/core/memory_allocator.h](../../executorch/runtime/core/memory_allocator.h) | Bump allocator | +| [../../executorch/runtime/core/device_allocator.h](../../executorch/runtime/core/device_allocator.h) | `DeviceAllocator` registry | +| [../../executorch/runtime/core/device_memory_buffer.h](../../executorch/runtime/core/device_memory_buffer.h) | RAII device buffer | +| [../../executorch/runtime/core/portable_type/device.h](../../executorch/runtime/core/portable_type/device.h) | `Device{CPU,CUDA}` | +| [../../executorch/runtime/core/tensor_shape_dynamism.h](../../executorch/runtime/core/tensor_shape_dynamism.h) | Shape dynamism enum | +| [../../executorch/runtime/backend/interface.h](../../executorch/runtime/backend/interface.h) | `BackendInterface` | +| [../../executorch/runtime/backend/interface.cpp](../../executorch/runtime/backend/interface.cpp) | Backend registry (16) | +| [../../executorch/runtime/kernel/operator_registry.h](../../executorch/runtime/kernel/operator_registry.h) | Kernel registry API | +| [../../executorch/runtime/kernel/operator_registry.cpp](../../executorch/runtime/kernel/operator_registry.cpp) | Kernel table limits | +| [../../executorch/runtime/kernel/thread_parallel_interface.h](../../executorch/runtime/kernel/thread_parallel_interface.h) | `parallel_for` | +| [../../executorch/runtime/platform/platform.h](../../executorch/runtime/platform/platform.h) | PAL | +| [../../executorch/exir/program/_program.py](../../executorch/exir/program/_program.py) | `to_edge`, managers | +| [../../executorch/exir/capture/_config.py](../../executorch/exir/capture/_config.py) | Compile/backend configs | +| [../../executorch/exir/memory_planning.py](../../executorch/exir/memory_planning.py) | Planning algorithms | +| [../../executorch/exir/banked_memory_planning.py](../../executorch/exir/banked_memory_planning.py) | Capacity-aware planning | +| [../../executorch/exir/dynamic_shape.py](../../executorch/exir/dynamic_shape.py) | `DynamicMemoryPlanningMode` | +| [../../executorch/exir/passes/sym_shape_eval_pass.py](../../executorch/exir/passes/sym_shape_eval_pass.py) | Upper-bound shape eval | +| [../../executorch/exir/passes/propagate_device_pass.py](../../executorch/exir/passes/propagate_device_pass.py) | h2d/d2h insertion | +| [../../executorch/exir/passes/_device_copy_ops_registry.py](../../executorch/exir/passes/_device_copy_ops_registry.py) | `_h2d_copy/_d2h_copy` ops | +| [../../executorch/exir/passes/quant_fusion_pass.py](../../executorch/exir/passes/quant_fusion_pass.py) | Q/DQ fusion | +| [../../executorch/exir/emit/_emitter.py](../../executorch/exir/emit/_emitter.py) | Emitter | +| [../../executorch/exir/backend/backend_api.py](../../executorch/exir/backend/backend_api.py) | `to_backend` | +| [../../executorch/exir/backend/partitioner.py](../../executorch/exir/backend/partitioner.py) | `Partitioner` | +| [../../executorch/exir/backend/backend_details.py](../../executorch/exir/backend/backend_details.py) | `BackendDetails` | +| [../../executorch/export](../../executorch/export) | Recipe export API | +| [../../executorch/kernels/README.md](../../executorch/kernels/README.md) | Kernel lib layout | +| [../../executorch/kernels/portable/functions.yaml](../../executorch/kernels/portable/functions.yaml) | Portable ops (209) | +| [../../executorch/kernels/portable/custom_ops.yaml](../../executorch/kernels/portable/custom_ops.yaml) | Portable custom ops | +| [../../executorch/kernels/optimized/optimized.yaml](../../executorch/kernels/optimized/optimized.yaml) | Optimized ops (23) | +| [../../executorch/kernels/optimized/cpu/op_fft_r2c.cpp](../../executorch/kernels/optimized/cpu/op_fft_r2c.cpp) | FFT kernel | +| [../../executorch/kernels/quantized/quantized.yaml](../../executorch/kernels/quantized/quantized.yaml) | Quantized ops (19) | +| [../../executorch/kernels/aten/functions.yaml](../../executorch/kernels/aten/functions.yaml) | ATen mode ops | +| [../../executorch/kernels/prim_ops/register_prim_ops.cpp](../../executorch/kernels/prim_ops/register_prim_ops.cpp) | Prim ops | +| [../../executorch/backends/xnnpack/README.md](../../executorch/backends/xnnpack/README.md) | XNNPACK backend | +| [../../executorch/backends/xnnpack/CMakeLists.txt](../../executorch/backends/xnnpack/CMakeLists.txt) | KleidiAI flag | +| [../../executorch/backends/xnnpack/runtime/XNNPACKBackend.cpp](../../executorch/backends/xnnpack/runtime/XNNPACKBackend.cpp) | Registration | +| [../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp](../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp) | Subgraph → runtime | +| [../../executorch/backends/xnnpack/runtime/XNNExecutor.cpp](../../executorch/backends/xnnpack/runtime/XNNExecutor.cpp) | Dynamic reshape/forward | +| [../../executorch/backends/xnnpack/quantizer/xnnpack_quantizer.py](../../executorch/backends/xnnpack/quantizer/xnnpack_quantizer.py) | XNNPACK quantizer | +| [../../executorch/backends/vulkan/README.md](../../executorch/backends/vulkan/README.md) | Vulkan backend | +| [../../executorch/backends/vulkan/runtime/VulkanBackend.cpp](../../executorch/backends/vulkan/runtime/VulkanBackend.cpp) | Vulkan delegate | +| [../../executorch/backends/vulkan/runtime/graph/ComputeGraph.h](../../executorch/backends/vulkan/runtime/graph/ComputeGraph.h) | Vulkan graph exec | +| [../../executorch/backends/webgpu/README.md](../../executorch/backends/webgpu/README.md) | WebGPU backend | +| [../../executorch/backends/webgpu/runtime/WebGPUBackend.cpp](../../executorch/backends/webgpu/runtime/WebGPUBackend.cpp) | Registration | +| [../../executorch/backends/qualcomm/README.md](../../executorch/backends/qualcomm/README.md) | QNN backend | +| [../../executorch/backends/qualcomm/runtime/QnnExecuTorchBackend.cpp](../../executorch/backends/qualcomm/runtime/QnnExecuTorchBackend.cpp) | Registration | +| [../../executorch/backends/qualcomm/runtime/QnnManager.h](../../executorch/backends/qualcomm/runtime/QnnManager.h) | Graph exec, shared mem | +| [../../executorch/backends/qualcomm/serialization/qc_schema.py](../../executorch/backends/qualcomm/serialization/qc_schema.py) | Chipset enum | +| [../../executorch/backends/qualcomm/genai_pipeline/genai_pipeline.py](../../executorch/backends/qualcomm/genai_pipeline/genai_pipeline.py) | QNN GenAI pipeline | +| [../../executorch/backends/mediatek/README.md](../../executorch/backends/mediatek/README.md) | MediaTek backend | +| [../../executorch/backends/mediatek/runtime/NeuronBackend.cpp](../../executorch/backends/mediatek/runtime/NeuronBackend.cpp) | Registration | +| [../../executorch/backends/samsung/README.md](../../executorch/backends/samsung/README.md) | Samsung backend | +| [../../executorch/backends/samsung/runtime/enn_backend.cpp](../../executorch/backends/samsung/runtime/enn_backend.cpp) | Registration | +| [../../executorch/backends/arm/README.md](../../executorch/backends/arm/README.md) | Arm TOSA/Ethos-U/VGF | +| [../../executorch/backends/arm/runtime/EthosUBackend.cpp](../../executorch/backends/arm/runtime/EthosUBackend.cpp) | Ethos-U delegate | +| [../../executorch/backends/arm/runtime/VGFBackend.cpp](../../executorch/backends/arm/runtime/VGFBackend.cpp) | VGF delegate | +| [../../executorch/backends/cortex_m/README.md](../../executorch/backends/cortex_m/README.md) | Cortex-M dialect | +| [../../executorch/backends/cadence/README.md](../../executorch/backends/cadence/README.md) | Cadence DSP | +| [../../executorch/backends/nxp/README.md](../../executorch/backends/nxp/README.md) | NXP Neutron | +| [../../executorch/backends/nxp/runtime/NeutronBackend.cpp](../../executorch/backends/nxp/runtime/NeutronBackend.cpp) | Registration | +| [../../executorch/backends/openvino/README.md](../../executorch/backends/openvino/README.md) | OpenVINO | +| [../../executorch/backends/openvino/runtime/OpenvinoBackend.cpp](../../executorch/backends/openvino/runtime/OpenvinoBackend.cpp) | Registration | +| [../../executorch/backends/apple/coreml/README.md](../../executorch/backends/apple/coreml/README.md) | CoreML backend | +| [../../executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm](../../executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm) | CoreML delegate | +| [../../executorch/backends/apple/metal/README.md](../../executorch/backends/apple/metal/README.md) | Metal (experimental) | +| [../../executorch/backends/apple/metal/runtime/metal_backend.cpp](../../executorch/backends/apple/metal/runtime/metal_backend.cpp) | Registration | +| [../../executorch/backends/apple/coreai/README.md](../../executorch/backends/apple/coreai/README.md) | Core AI (WIP) | +| [../../executorch/backends/mlx/README.md](../../executorch/backends/mlx/README.md) | MLX backend | +| [../../executorch/backends/mlx/runtime/MLXBackend.cpp](../../executorch/backends/mlx/runtime/MLXBackend.cpp) | MLX runtime | +| [../../executorch/backends/mlx/serialization](../../executorch/backends/mlx/serialization) | MLX bytecode | +| [../../executorch/backends/cuda/cuda_backend.py](../../executorch/backends/cuda/cuda_backend.py) | CUDA AOTI export | +| [../../executorch/backends/cuda/runtime/cuda_backend.cpp](../../executorch/backends/cuda/runtime/cuda_backend.cpp) | CUDA delegate, CUDA graphs | +| [../../executorch/backends/cuda/runtime/cuda_allocator.h](../../executorch/backends/cuda/runtime/cuda_allocator.h) | CUDA `DeviceAllocator` | +| [../../executorch/backends/cuda/runtime/cuda_mutable_state.h](../../executorch/backends/cuda/runtime/cuda_mutable_state.h) | Per-session state | +| [../../executorch/backends/cuda/runtime/shims/int4mm.cuh](../../executorch/backends/cuda/runtime/shims/int4mm.cuh) | int4 GEMM | +| [../../executorch/backends/cuda/quantize_op_dispatch/int4_dispatch.py](../../executorch/backends/cuda/quantize_op_dispatch/int4_dispatch.py) | Quant dispatch | +| [../../executorch/backends/cuda/rocm.md](../../executorch/backends/cuda/rocm.md) | ROCm status | +| [../../executorch/backends/aoti/README.md](../../executorch/backends/aoti/README.md) | AOTI common lib | +| [../../executorch/backends/native/preprocess.py](../../executorch/backends/native/preprocess.py) | Native delegate | +| [../../executorch/backends/native/runtime/Program.h](../../executorch/backends/native/runtime/Program.h) | Native runtime | +| [../../executorch/backends/example/README.md](../../executorch/backends/example/README.md) | Example delegate | +| [../../executorch/backends/test/README.md](../../executorch/backends/test/README.md) | Backend test harness | +| [../../executorch/backends/transforms/convert_conv1d_to_conv2d_pass.py](../../executorch/backends/transforms/convert_conv1d_to_conv2d_pass.py) | Conv1d transform | +| [../../executorch/extension/module/module.h](../../executorch/extension/module/module.h) | `Module` | +| [../../executorch/extension/data_loader/mmap_data_loader.h](../../executorch/extension/data_loader/mmap_data_loader.h) | mmap loader | +| [../../executorch/extension/flat_tensor/flat_tensor_data_map.h](../../executorch/extension/flat_tensor/flat_tensor_data_map.h) | `.ptd` loader | +| [../../executorch/extension/threadpool/threadpool.h](../../executorch/extension/threadpool/threadpool.h) | Threadpool | +| [../../executorch/extension/threadpool/threadpool.cpp](../../executorch/extension/threadpool/threadpool.cpp) | Impl | +| [../../executorch/extension/threadpool/cpuinfo_utils.cpp](../../executorch/extension/threadpool/cpuinfo_utils.cpp) | Performant cores | +| [../../executorch/extension/llm/runner/README.md](../../executorch/extension/llm/runner/README.md) | Runner framework | +| [../../executorch/extension/llm/runner/irunner.h](../../executorch/extension/llm/runner/irunner.h) | `IRunner`, `GenerationConfig` | +| [../../executorch/extension/llm/runner/text_llm_runner.h](../../executorch/extension/llm/runner/text_llm_runner.h) | `TextLLMRunner` | +| [../../executorch/extension/llm/runner/text_llm_runner.cpp](../../executorch/extension/llm/runner/text_llm_runner.cpp) | generate() | +| [../../executorch/extension/llm/runner/text_prefiller.h](../../executorch/extension/llm/runner/text_prefiller.h) | Prefill | +| [../../executorch/extension/llm/runner/text_token_generator.h](../../executorch/extension/llm/runner/text_token_generator.h) | Decode loop | +| [../../executorch/extension/llm/runner/text_decoder_runner.h](../../executorch/extension/llm/runner/text_decoder_runner.h) | Decoder step | +| [../../executorch/extension/llm/runner/text_decoder_runner.cpp](../../executorch/extension/llm/runner/text_decoder_runner.cpp) | step impl | +| [../../executorch/extension/llm/runner/io_manager/io_manager.h](../../executorch/extension/llm/runner/io_manager/io_manager.h) | `IOManager` | +| [../../executorch/extension/llm/runner/multimodal_runner.h](../../executorch/extension/llm/runner/multimodal_runner.h) | `MultimodalRunner` | +| [../../executorch/extension/llm/runner/multimodal_prefiller.cpp](../../executorch/extension/llm/runner/multimodal_prefiller.cpp) | Encoder invocation | +| [../../executorch/extension/llm/runner/multimodal_input.h](../../executorch/extension/llm/runner/multimodal_input.h) | Input variant | +| [../../executorch/extension/llm/runner/image.h](../../executorch/extension/llm/runner/image.h) | `Image` | +| [../../executorch/extension/llm/runner/audio.h](../../executorch/extension/llm/runner/audio.h) | `Audio`, `RawAudio` | +| [../../executorch/extension/llm/runner/constants.h](../../executorch/extension/llm/runner/constants.h) | Method name conventions | +| [../../executorch/extension/llm/runner/llm_session.h](../../executorch/extension/llm/runner/llm_session.h) | `LLMSession`, `LLMEngine` | +| [../../executorch/extension/llm/runner/llm_runner_helper.h](../../executorch/extension/llm/runner/llm_runner_helper.h) | Factories, tokenizer loading | +| [../../executorch/extension/llm/runner/stats.h](../../executorch/extension/llm/runner/stats.h) | `Stats` | +| [../../executorch/extension/llm/runner/pybindings.cpp](../../executorch/extension/llm/runner/pybindings.cpp) | Runner pybindings | +| [../../executorch/extension/llm/sampler/sampler.h](../../executorch/extension/llm/sampler/sampler.h) | `Sampler` | +| [../../executorch/extension/llm/sampler/logit_processor.h](../../executorch/extension/llm/sampler/logit_processor.h) | `LogitProcessor` | +| [../../executorch/extension/llm/batching/runner.h](../../executorch/extension/llm/batching/runner.h) | Batching runner/session | +| [../../executorch/extension/llm/batching/decode_first_scheduler.h](../../executorch/extension/llm/batching/decode_first_scheduler.h) | Scheduler impl | +| [../../executorch/extension/llm/cache/cache.h](../../executorch/extension/llm/cache/cache.h) | Cache interfaces | +| [../../executorch/extension/llm/cache/cell_cache.h](../../executorch/extension/llm/cache/cell_cache.h) | Cell (paged-like) cache | +| [../../executorch/extension/llm/cache/sequence_cache.h](../../executorch/extension/llm/cache/sequence_cache.h) | Flat/Ring policies | +| [../../executorch/extension/llm/cache/update_and_attend.py](../../executorch/extension/llm/cache/update_and_attend.py) | `kvcache::update_and_attend` | +| [../../executorch/extension/llm/custom_ops/custom_ops.py](../../executorch/extension/llm/custom_ops/custom_ops.py) | Custom op defs | +| [../../executorch/extension/llm/custom_ops/op_sdpa.cpp](../../executorch/extension/llm/custom_ops/op_sdpa.cpp) | SDPA kernel | +| [../../executorch/extension/llm/custom_ops/op_update_cache.cpp](../../executorch/extension/llm/custom_ops/op_update_cache.cpp) | KV update kernel | +| [../../executorch/extension/llm/custom_ops/op_tile_crop.cpp](../../executorch/extension/llm/custom_ops/op_tile_crop.cpp) | Image tile crop | +| [../../executorch/extension/llm/custom_ops/op_fallback.py](../../executorch/extension/llm/custom_ops/op_fallback.py) | Fallback op | +| [../../executorch/extension/llm/export/export_llm.py](../../executorch/extension/llm/export/export_llm.py) | LLM export CLI | +| [../../executorch/extension/llm/export/builder.py](../../executorch/extension/llm/export/builder.py) | `LLMEdgeManager` | +| [../../executorch/extension/llm/export/config/llm_config.py](../../executorch/extension/llm/export/config/llm_config.py) | `LlmConfig`, `ModelType` | +| [../../executorch/extension/llm/export/partitioner_lib.py](../../executorch/extension/llm/export/partitioner_lib.py) | Backend partitioners for LLM | +| [../../executorch/extension/llm/export/quantizer_lib.py](../../executorch/extension/llm/export/quantizer_lib.py) | PT2E quantizers | +| [../../executorch/extension/llm/apple](../../executorch/extension/llm/apple) | `ExecuTorchLLM` Swift | +| [../../executorch/extension/asr/runner/seq2seq_runner.h](../../executorch/extension/asr/runner/seq2seq_runner.h) | ASR seq2seq | +| [../../executorch/extension/audio/mel_spectrogram.py](../../executorch/extension/audio/mel_spectrogram.py) | Mel front end | +| [../../executorch/extension/image/image_processor.h](../../executorch/extension/image/image_processor.h) | Image preprocessing | +| [../../executorch/extension/android/README.md](../../executorch/extension/android/README.md) | Android bindings | +| [../../executorch/extension/android/jni/jni_layer_llama.cpp](../../executorch/extension/android/jni/jni_layer_llama.cpp) | LLM JNI | +| [../../executorch/extension/apple/ExecuTorch/Exported/ExecuTorchModule.h](../../executorch/extension/apple/ExecuTorch/Exported/ExecuTorchModule.h) | Apple bindings | +| [../../executorch/extension/pybindings/pybindings.cpp](../../executorch/extension/pybindings/pybindings.cpp) | Python bindings | +| [../../executorch/extension/wasm/wasm_bindings.cpp](../../executorch/extension/wasm/wasm_bindings.cpp) | WASM bindings | +| [../../executorch/extension/wasm/tokenizers](../../executorch/extension/wasm/tokenizers) | WASM tokenizers | +| [../../executorch/devtools/etdump/etdump_flatcc.h](../../executorch/devtools/etdump/etdump_flatcc.h) | ETDump | +| [../../executorch/devtools/inspector/_inspector.py](../../executorch/devtools/inspector/_inspector.py) | Inspector | +| [../../executorch/devtools/bundled_program/bundled_program.h](../../executorch/devtools/bundled_program/bundled_program.h) | Bundled program | +| [../../executorch/devtools/size_analysis_tool](../../executorch/devtools/size_analysis_tool) | Size analysis | +| [../../executorch/examples/models/llama/README.md](../../executorch/examples/models/llama/README.md) | Llama example + perf | +| [../../executorch/examples/models/llama/export_llama_lib.py](../../executorch/examples/models/llama/export_llama_lib.py) | Export pipeline | +| [../../executorch/examples/models/llama/model_args.py](../../executorch/examples/models/llama/model_args.py) | `ModelArgs` | +| [../../executorch/examples/models/llama/attention.py](../../executorch/examples/models/llama/attention.py) | `KVCache`, attention | +| [../../executorch/examples/models/llama/static_attention.py](../../executorch/examples/models/llama/static_attention.py) | Static attention (Python) | +| [../../executorch/examples/models/llama/runner/static_attention_io_manager.h](../../executorch/examples/models/llama/runner/static_attention_io_manager.h) | Static attention (C++) | +| [../../executorch/examples/models/llama/runner/runner.cpp](../../executorch/examples/models/llama/runner/runner.cpp) | Llama runner factory | +| [../../executorch/examples/models/llama/source_transformation/quantize.py](../../executorch/examples/models/llama/source_transformation/quantize.py) | Quant modes | +| [../../executorch/examples/models/llama/source_transformation/custom_kv_cache.py](../../executorch/examples/models/llama/source_transformation/custom_kv_cache.py) | KV cache variants | +| [../../executorch/examples/models/llama/source_transformation/sdpa.py](../../executorch/examples/models/llama/source_transformation/sdpa.py) | SDPA variants | +| [../../executorch/examples/models/llama/source_transformation/apply_spin_quant_r1_r2.py](../../executorch/examples/models/llama/source_transformation/apply_spin_quant_r1_r2.py) | SpinQuant | +| [../../executorch/examples/models/llava/README.md](../../executorch/examples/models/llava/README.md) | LLaVA | +| [../../executorch/examples/models/llava/export_llava.py](../../executorch/examples/models/llava/export_llava.py) | LLaVA export | +| [../../executorch/examples/models/llama3_2_vision/preprocess/preprocess.h](../../executorch/examples/models/llama3_2_vision/preprocess/preprocess.h) | Vision preprocess C++ | +| [../../executorch/examples/models/smolvlm/convert_weights.py](../../executorch/examples/models/smolvlm/convert_weights.py) | SmolVLM | +| [../../executorch/examples/models/gemma3/e2e_runner.cpp](../../executorch/examples/models/gemma3/e2e_runner.cpp) | Gemma3 VLM runner | +| [../../executorch/examples/models/voxtral_tts/README.md](../../executorch/examples/models/voxtral_tts/README.md) | Voxtral TTS | +| [../../executorch/examples/models/voxtral_tts/export_voxtral_tts.py](../../executorch/examples/models/voxtral_tts/export_voxtral_tts.py) | TTS export | +| [../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.h](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.h) | TTS runner API | +| [../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.cpp](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.cpp) | TTS runner impl | +| [../../executorch/examples/models/voxtral_tts/wav_writer.h](../../executorch/examples/models/voxtral_tts/wav_writer.h) | WAV output | +| [../../executorch/examples/models/supertonic/README.md](../../executorch/examples/models/supertonic/README.md) | Supertonic TTS (MLX) | +| [../../executorch/examples/models/supertonic/runtime/supertonic_runner.cpp](../../executorch/examples/models/supertonic/runtime/supertonic_runner.cpp) | Supertonic runner | +| [../../executorch/examples/models/supertonic/model/vocoder.py](../../executorch/examples/models/supertonic/model/vocoder.py) | Vocoder | +| [../../executorch/examples/models/voxtral_realtime/README.md](../../executorch/examples/models/voxtral_realtime/README.md) | Streaming ASR | +| [../../executorch/examples/models/voxtral/README.md](../../executorch/examples/models/voxtral/README.md) | Audio LLM via optimum | +| [../../executorch/examples/models/whisper/README.md](../../executorch/examples/models/whisper/README.md) | Whisper runner | +| [../../executorch/examples/models/parakeet/README.md](../../executorch/examples/models/parakeet/README.md) | Parakeet TDT | +| [../../executorch/examples/models/silero_vad/README.md](../../executorch/examples/models/silero_vad/README.md) | VAD | +| [../../executorch/examples/models/silero_vad/stream_main.cpp](../../executorch/examples/models/silero_vad/stream_main.cpp) | Streaming VAD | +| [../../executorch/examples/models/sortformer/README.md](../../executorch/examples/models/sortformer/README.md) | Diarization | +| [../../executorch/examples/models/moshi/mimi/test_mimi.py](../../executorch/examples/models/moshi/mimi/test_mimi.py) | Mimi codec test | +| [../../executorch/examples/models/muse_glimmer/README.md](../../executorch/examples/models/muse_glimmer/README.md) | 30B VLM + serving | +| [../../executorch/examples/models/muse-glimmer/runtime/engine/muse_glimmer_engine.h](../../executorch/examples/models/muse-glimmer/runtime/engine/muse_glimmer_engine.h) | `LLMEngine` impl | +| [../../executorch/examples/models/qwen3_5_moe/qwen35_moe_engine.h](../../executorch/examples/models/qwen3_5_moe/qwen35_moe_engine.h) | `LLMEngine` impl | +| [../../executorch/examples/models/gemma4_31b/gemma4_31b_engine.h](../../executorch/examples/models/gemma4_31b/gemma4_31b_engine.h) | `LLMEngine` impl | +| [../../executorch/examples/models/stable_diffusion/model.py](../../executorch/examples/models/stable_diffusion/model.py) | SD model def | +| [../../executorch/examples/models/stable_diffusion_3_5_large/export_sd3_5_large.py](../../executorch/examples/models/stable_diffusion_3_5_large/export_sd3_5_large.py) | SD3.5 export | +| [../../executorch/examples/llm_server/README.md](../../executorch/examples/llm_server/README.md) | OpenAI server | +| [../../executorch/examples/llm_server/cpp/worker_loop.h](../../executorch/examples/llm_server/cpp/worker_loop.h) | Worker loop | +| [../../executorch/examples/llm_server/cpp/worker_prefill_plan.h](../../executorch/examples/llm_server/cpp/worker_prefill_plan.h) | Prefill planning | +| [../../executorch/examples/qualcomm/README.md](../../executorch/examples/qualcomm/README.md) | QNN examples | +| [../../executorch/examples/qualcomm/oss_scripts/llama/llama.py](../../executorch/examples/qualcomm/oss_scripts/llama/llama.py) | QNN LLM export | +| [../../executorch/examples/qualcomm/oss_scripts/llama/runner/kv_manager.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/kv_manager.h) | QNN KV manager | +| [../../executorch/examples/qualcomm/oss_scripts/llama/runner/rpc_mem.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/rpc_mem.h) | rpcmem shared buffers | +| [../../executorch/examples/qualcomm/oss_scripts/moshi/qnn_mimi_decoder_runner.cpp](../../executorch/examples/qualcomm/oss_scripts/moshi/qnn_mimi_decoder_runner.cpp) | Mimi on QNN | +| [../../executorch/examples/qualcomm/oss_scripts/dit.py](../../executorch/examples/qualcomm/oss_scripts/dit.py) | DiT on QNN | +| [../../executorch/examples/apple/coreml/README.md](../../executorch/examples/apple/coreml/README.md) | CoreML examples | +| [../../executorch/examples/arm/README.md](../../executorch/examples/arm/README.md) | Arm examples | +| [../../executorch/examples/mediatek/README.md](../../executorch/examples/mediatek/README.md) | MediaTek examples | +| [../../executorch/examples/samsung](../../executorch/examples/samsung) | Samsung examples | +| [../../executorch/examples/nxp](../../executorch/examples/nxp) | NXP examples | +| [../../executorch/examples/openvino](../../executorch/examples/openvino) | OpenVINO examples | +| [../../executorch/examples/vulkan](../../executorch/examples/vulkan) | Vulkan examples | +| [../../executorch/examples/xnnpack/README.md](../../executorch/examples/xnnpack/README.md) | XNNPACK examples | +| [../../executorch/examples/cuda/README.md](../../executorch/examples/cuda/README.md) | ROCm/AOTI example | +| [../../executorch/examples/portable/executor_runner/executor_runner.cpp](../../executorch/examples/portable/executor_runner/executor_runner.cpp) | Generic runner | +| [../../executorch/examples/portable/custom_ops](../../executorch/examples/portable/custom_ops) | Custom op example | +| [../../executorch/examples/selective_build](../../executorch/examples/selective_build) | Selective build example | +| [../../executorch/examples/arduino/README.md](../../executorch/examples/arduino/README.md) | Arduino | +| [../../executorch/examples/arduino/arduino_pal.cpp](../../executorch/examples/arduino/arduino_pal.cpp) | Arduino PAL | +| [../../executorch/examples/espressif/README.md](../../executorch/examples/espressif/README.md) | ESP32 | +| [../../executorch/examples/raspberry_pi/pico2](../../executorch/examples/raspberry_pi/pico2) | Pico 2 | +| [../../executorch/examples/riscv/README.md](../../executorch/examples/riscv/README.md) | RISC-V | +| [../../executorch/examples/zephyr](../../executorch/examples/zephyr) | Zephyr | +| [../../executorch/examples/wasm](../../executorch/examples/wasm) | WASM | +| [../../executorch/examples/llm_pte_finetuning](../../executorch/examples/llm_pte_finetuning) | On-device finetuning | +| [../../executorch/examples/demo-apps/react-native](../../executorch/examples/demo-apps/react-native) | RN demo | +| [../../executorch/zephyr](../../executorch/zephyr) | Zephyr module | +| [../../executorch/torch_pin.py](../../executorch/torch_pin.py) | PyTorch pin | +| [../../executorch/docs/source/backends-overview.md](../../executorch/docs/source/backends-overview.md) | Backend matrix | +| [../../executorch/docs/source/backends/xnnpack/xnnpack-overview.md](../../executorch/docs/source/backends/xnnpack/xnnpack-overview.md) | XNNPACK platforms | +| [../../executorch/docs/source/backends/cuda/cuda-overview.md](../../executorch/docs/source/backends/cuda/cuda-overview.md) | CUDA docs | +| [../../executorch/docs/source/backends/mlx/mlx-overview.md](../../executorch/docs/source/backends/mlx/mlx-overview.md) | MLX docs | +| [../../executorch/docs/source/backends/template](../../executorch/docs/source/backends/template) | Backend doc template | +| [../../executorch/docs/source/kernel-library-selective-build.md](../../executorch/docs/source/kernel-library-selective-build.md) | Selective build | +| [../../executorch/docs/source/kernel-library-custom-aten-kernel.md](../../executorch/docs/source/kernel-library-custom-aten-kernel.md) | Custom kernels | +| [../../executorch/docs/source/compiler-memory-planning.md](../../executorch/docs/source/compiler-memory-planning.md) | Memory planning | +| [../../executorch/docs/source/compiler-custom-compiler-passes.md](../../executorch/docs/source/compiler-custom-compiler-passes.md) | Custom passes | +| [../../executorch/docs/source/pte-file-format.md](../../executorch/docs/source/pte-file-format.md) | PTE format | +| [../../executorch/docs/source/ptd-file-format.md](../../executorch/docs/source/ptd-file-format.md) | PTD format | +| [../../executorch/docs/source/runtime-overview.md](../../executorch/docs/source/runtime-overview.md) | Runtime overview | +| [../../executorch/docs/source/getting-started-architecture.md](../../executorch/docs/source/getting-started-architecture.md) | Architecture | +| [../../executorch/docs/source/using-executorch-building-from-source.md](../../executorch/docs/source/using-executorch-building-from-source.md) | Build options/presets | +| [../../executorch/docs/source/using-executorch-android.md](../../executorch/docs/source/using-executorch-android.md) | Android AAR | +| [../../executorch/docs/source/using-executorch-ios.md](../../executorch/docs/source/using-executorch-ios.md) | iOS xcframeworks | +| [../../executorch/docs/source/llm/export-llm.md](../../executorch/docs/source/llm/export-llm.md) | LLM export doc | +| [../../executorch/docs/source/llm/export-llm-optimum.md](../../executorch/docs/source/llm/export-llm-optimum.md) | optimum-executorch | +| [../../executorch/docs/source/etdump.md](../../executorch/docs/source/etdump.md) | ETDump | +| [../../executorch/docs/source/raspberry_pi_llama_tutorial.md](../../executorch/docs/source/raspberry_pi_llama_tutorial.md) | RPi tutorial | +| [../../executorch/docs/source/zephyr_alif_tutorial.md](../../executorch/docs/source/zephyr_alif_tutorial.md) | Zephyr/Alif tutorial | diff --git a/docs/engines/llama_cpp.md b/docs/engines/llama_cpp.md new file mode 100644 index 000000000..f91cb36fb --- /dev/null +++ b/docs/engines/llama_cpp.md @@ -0,0 +1,558 @@ +# llama.cpp (ggml-org/llama.cpp) — Engine Analysis + +| Field | Value | +|---|---| +| Local path | `/data/zhoutaichang/embedding_infer/llama.cpp` | +| Remote | `https://github.com/ggml-org/llama.cpp.git` (origin) | +| Branch | `master` | +| HEAD | `e71b80510c848c00175924ecf3c40333ccae8eb5` (2026-09-07, "Revert CUDA: size routed MoE MMQ N-tiles ... (#28551)") | +| Dirty status | clean (`git status --porcelain` empty) | +| Submodules | none declared (`git submodule status` prints nothing); third-party code is vendored under `vendor/` (cpp-httplib, miniaudio, nlohmann, stb, sheredom, hash) | +| Analysis date | 2026-09-08 | +| Method | read-only source/doc inspection; no builds, no runs; evidence labels: [Verified] / [Inferred] / [Proposal] / [Unknown] | + +Note: this checkout is far ahead of common public knowledge about llama.cpp. Several features described here (mtmd audio generation, Hexagon/HTP backend, backend samplers, `llama_model_init_from_user`, DSv4/MSA KV caches, unified `llama` app binary) may not exist in older releases. Everything below is grounded in this tree. + +## 0. Summary + +- [Verified] llama.cpp is a C/C++ inference engine built on the `ggml` tensor library. Two layers: `ggml` (tensor/graph/backends/scheduler, `../../llama.cpp/ggml/`) and `libllama` (model loading, KV/memory, graph building, sampling, tokenization, `../../llama.cpp/src/`), plus `libmtmd` (multimodal encoders + audio generation, `../../llama.cpp/tools/mtmd/`) and `common/` (CLI args, chat templates, speculative decoding). +- [Verified] It is primarily an **execution runtime + library**; the "serving stack" is a separate tool, `llama-server` (`../../llama.cpp/tools/server/`), which implements slot-based continuous batching, an OpenAI/Anthropic-compatible HTTP API, SSE streaming, a router mode that spawns child server processes, and prompt caching. +- [Verified] 18 ggml backend directories exist in-tree: `blas, cann, cpu, cuda, et, hexagon, hip, metal, musa, opencl, openvino, rpc, sycl, virtgpu, vulkan, webgpu, zdnn, zendnn` (`../../llama.cpp/ggml/src/`). All are registered via `ggml_backend_registry` (`../../llama.cpp/ggml/src/ggml-backend-reg.cpp`) and can be built as dynamically loaded plugins (`GGML_BACKEND_DL`). +- [Verified] Heterogeneous execution: `ggml_backend_sched` (`../../llama.cpp/ggml/src/ggml-backend.cpp`) assigns each graph node to a backend (weights location, `supports_op`, `offload_op`), splits the graph, inserts inter-backend copies, and supports pipeline parallelism with up to `GGML_SCHED_MAX_COPIES=4` in-flight copies + events. Unsupported ops fall back to the CPU backend (last-priority backend). +- [Verified] Quantization: 43 `ggml_type` ids incl. Q4_0/Q4_1/Q5_x/Q8_0, K-quants (Q2_K..Q6_K), IQ1..IQ4 variants, TQ1_0/TQ2_0 ternary, MXFP4, NVFP4, Q1_0/Q2_0 (`../../llama.cpp/ggml/include/ggml.h#L390`). Dequantization is done on the fly inside matmul kernels (CPU `vec_dot` over quantized blocks after quantizing activations to `vec_dot_type`; CUDA `mmq`/`mmvq`). +- [Verified] Edge CPU acceleration: runtime weight repacking (`GGML_CPU_REPACK`, on by default) into interleaved Q4_0/Q4_K/Q2_K/Q5_K/Q6_K/Q8_0/IQ4_NL/MXFP4 layouts (`../../llama.cpp/ggml/src/ggml-cpu/repack.cpp`), optional Arm KleidiAI microkernels (`GGML_CPU_KLEIDIAI`, `../../llama.cpp/ggml/src/ggml-cpu/kleidiai/`), AMX on x86, per-arch SIMD in `ggml-cpu/arch/{arm,x86,riscv,loongarch,powerpc,s390,wasm}`. +- [Verified] Mobile/edge GPU/NPU backends present: Metal (Apple, default ON on APPLE), Vulkan, OpenCL (Qualcomm Adreno-optimized, prebuilt Adreno binary kernel library optional), Hexagon HTP (Snapdragon NPU, `../../llama.cpp/ggml/src/ggml-hexagon/`), CANN (Ascend), OpenVINO (Intel NPU, "in progress"), WebGPU, ET (RISC-V manycore). No CoreML, no NNAPI, no QNN backend in tree. +- [Verified] Model runner: GGUF format only (converted by `convert_hf_to_gguf.py` + `conversion/*.py` + `gguf-py`). `llama_model_load_from_file` -> `llama_init_from_model` -> `llama_decode`/`llama_encode`. Graph is rebuilt per ubatch unless `llm_graph_result::can_reuse()` matches (graph reuse), and compute buffers are reserved up front (`llama_context::sched_reserve`). +- [Verified] KV/memory abstraction `llama_memory_i` has many implementations: unified/per-stream KV cache, iSWA (two caches), DSA/DSA-iSWA, MSA, DSv4, recurrent (Mamba/RWKV, with rollback snapshots), hybrid, hybrid-iSWA, hybrid-idx (`../../llama.cpp/src/llama-memory.h`, `../../llama.cpp/src/llama-model.cpp#L2232`). KV cells are a ring buffer with per-cell sequence bitsets; no paged attention. +- [Verified] TTS is first-class at this commit: `tools/tts` (`llama-tts`) drives a 3-stage pipeline (backbone -> code predictor / flow net -> code2wav / Mimi decoder) through `mtmd` experimental audio-generation API. Supported pipelines: **Qwen3-TTS** and **Pocket-TTS (Kyutai)**; legacy **WavTokenizer decoder** arch remains in `libllama` (OuteTTS-era). Output is WAV to file, non-streaming in the CLI, but the API is frame-stepped (`step_gen`) so streaming is feasible. +- [Verified] Audio input (ASR/audio understanding) is supported through mtmd audio encoders (Whisper-style, Conformer/Parakeet, Ultravox, Qwen2/3-Audio, Voxtral, Gemma-4 audio, etc.). Server exposes `/v1/audio/transcriptions`. Mel preprocessing is a whisper.cpp-derived FFT in `mtmd-audio.cpp`. +- [Verified] Vision: `clip.cpp` in mtmd implements ~60 projector types (SigLIP/CLIP ViTs, Qwen-VL mergers, Gemma3/4, Pixtral, InternVL, Kimi-VL, MobileNetV5 for Gemma3n, etc.), with image preprocessing (Pillow-compatible resize, LLaVA-UHD slicing) and video (ffmpeg subprocess). +- [Verified] No VLA support: no action heads, no proprioceptive input path, no diffusion/flow policy heads for robotics. The only generic "numeric input" path is `llama_batch.embd` (pre-computed input embeddings) and `mtmd` encoder outputs. Diffusion **language** models (Dream/LLaDA) exist (`../../llama.cpp/examples/diffusion/`), and Pocket-TTS contains a flow-matching net in mtmd, which are the closest existing building blocks. +- [Verified] Application boundaries: C API (`include/llama.h`, `tools/mtmd/mtmd.h`), unified `llama` binary (`app/`), `llama-cli`, `llama-server`, `llama-mtmd-cli`, `llama-tts`, Android JNI/Kotlin sample (`examples/llama.android`), iOS SwiftUI + XCFramework (`examples/llama.swiftui`, `build-xcframework.sh`), RPC server for distributed backends. +- [Unknown] The repo publishes no binary-size numbers; footprint must be measured. + +## 1. Architecture overview + +```mermaid +graph TD + subgraph Apps["Applications / boundaries"] + CLI[llama-cli / llama app
tools/cli, app/] + SRV[llama-server
tools/server] + TTS[llama-tts
tools/tts] + MTMDCLI[llama-mtmd-cli] + MOB[Android JNI / iOS Swift
examples/llama.android, llama.swiftui] + end + subgraph Common["common/ (C++ helpers)"] + ARG[arg.cpp params] + CHAT[chat.cpp + jinja engine] + SPEC[speculative.cpp] + SAMP[sampling.cpp] + end + subgraph MTMD["libmtmd (tools/mtmd)"] + MTMDAPI[mtmd.h / mtmd-helper.h] + CLIP[clip.cpp encoders + projectors] + AUD[mtmd-audio.cpp mel/FFT] + IMG[mtmd-image.cpp preprocessing] + GEN[mtmd-helper-gen.cpp
TTS pipelines] + end + subgraph LLAMA["libllama (src/)"] + LOAD[llama-model-loader.cpp
GGUF + mmap] + MODEL[llama-model.cpp + src/models/*
tensors + graph builders] + CTX[llama-context.cpp
decode/encode, sched, outputs] + BATCH[llama-batch.cpp
ubatch splitting] + MEM[llama-memory-*.cpp / llama-kv-cache*.cpp] + GRAPH[llama-graph.cpp
llm_graph_context] + VOCAB[llama-vocab.cpp tokenizers] + SMPL[llama-sampler.cpp] + end + subgraph GGML["ggml (ggml/)"] + GG[ggml.c tensors/ops/graphs] + SCHED[ggml-backend.cpp
ggml_backend_sched] + ALLOC[ggml-alloc.c gallocr] + REG[ggml-backend-reg.cpp registry + DL] + BE[backends: cpu cuda metal vulkan opencl hexagon sycl hip cann rpc webgpu ...] + end + CLI --> Common --> LLAMA + SRV --> Common + SRV --> MTMD + TTS --> MTMD + MTMDCLI --> MTMD + MOB --> LLAMA + MTMD --> LLAMA + MTMD --> GGML + LLAMA --> GGML + SCHED --> ALLOC + SCHED --> BE + REG --> BE +``` + +Key modules: + +| Module | Path | Responsibility | Key symbols | +|---|---|---|---| +| ggml core | `../../llama.cpp/ggml/src/ggml.c`, `../../llama.cpp/ggml/include/ggml.h` | tensor struct, op enum (102 ops), graph build | `ggml_tensor`, `enum ggml_op`, `ggml_build_forward_expand`, `ggml_custom_4d` | +| Backend API | `../../llama.cpp/ggml/include/ggml-backend.h`, `../../llama.cpp/ggml/src/ggml-backend-impl.h` | device/buffer/backend vtables | `ggml_backend_device_i`, `ggml_backend_i`, `ggml_backend_reg_i`, `ggml_backend_dev_props` | +| Backend registry | `../../llama.cpp/ggml/src/ggml-backend-reg.cpp` | compile-time registration, dynamic loading | `ggml_backend_registry`, `ggml_backend_load_all`, `ggml_backend_load_best` | +| Scheduler | `../../llama.cpp/ggml/src/ggml-backend.cpp` | node->backend assignment, splits, copies, pipeline parallel | `ggml_backend_sched`, `ggml_backend_sched_split_graph`, `ggml_backend_sched_compute_splits` | +| Graph allocator | `../../llama.cpp/ggml/src/ggml-alloc.c` | compute buffer planning (in-place reuse, chunked vbuffers) | `ggml_gallocr`, `ggml_dyn_tallocr`, `ggml_backend_alloc_ctx_tensors` | +| CPU backend | `../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c`, `ggml-cpu.cpp`, `ops.cpp` | threadpool, op kernels, extra buffer types | `ggml_threadpool`, `ggml_graph_compute`, `type_traits_cpu`, `ggml_backend_cpu_device_supports_op` | +| GPU/NPU backends | `../../llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu`, `../../llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp`, `../../llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp`, `../../llama.cpp/ggml/src/ggml-opencl/ggml-opencl.cpp`, `../../llama.cpp/ggml/src/ggml-hexagon/ggml-hexagon.cpp` | per-backend kernels | `ggml_backend_*_device_supports_op`, `*_reg()` | +| RPC backend | `../../llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp`, `../../llama.cpp/tools/rpc/rpc-server.cpp` | remote device over sockets | `enum rpc_cmd` | +| Model loader | `../../llama.cpp/src/llama-model-loader.cpp`, `../../llama.cpp/src/llama-mmap.cpp` | GGUF parse, splits, mmap/mlock, tensor data upload | `llama_model_loader`, `load_all_data`, `init_mappings` | +| Model | `../../llama.cpp/src/llama-model.cpp`, `../../llama.cpp/src/models/models.h` (+ one `.cpp` per arch in `../../llama.cpp/src/models/`) | hparams/tensors per arch, buffer placement, `create_memory`, `build_graph` | `llama_model::load_tensors`, `make_cpu_buft_list`, `make_gpu_buft_list`, `llama_model::create_memory` | +| Context | `../../llama.cpp/src/llama-context.cpp` | backends/sched creation, reserve, decode/encode loop, outputs, state save/load | `llama_context::decode`, `process_ubatch`, `sched_reserve`, `graph_compute` | +| Batch | `../../llama.cpp/src/llama-batch.cpp` | validate batch, split into ubatches | `llama_batch_allocr::split_simple/split_equal/split_seq` | +| Memory/KV | `../../llama.cpp/src/llama-memory.h`, `../../llama.cpp/src/llama-kv-cache.cpp`, `llama-kv-cells.h`, `llama-memory-recurrent.cpp` | KV storage, slot finding, masks, shifts, state IO | `llama_memory_i`, `llama_kv_cache::find_slot`, `apply_ubatch`, `llama_kv_cells` | +| Graph builder | `../../llama.cpp/src/llama-graph.cpp` | shared building blocks, graph inputs, reuse checks | `llm_graph_context::build_attn/build_ffn/build_moe_ffn/build_rs`, `llm_graph_result::can_reuse` | +| Vocab | `../../llama.cpp/src/llama-vocab.cpp` | tokenizers SPM/BPE/WPM/UGM/RWKV/PLaMo2, detokenize | `llm_tokenizer_*_session`, `llama_vocab::impl` | +| Sampling | `../../llama.cpp/src/llama-sampler.cpp` | CPU sampler chain + backend (in-graph) samplers | `llama_sampler_chain_*`, `llama_set_sampler` | +| Multimodal | `../../llama.cpp/tools/mtmd/mtmd.cpp`, `clip.cpp`, `mtmd-audio.cpp`, `mtmd-image.cpp`, `mtmd-helper.cpp` | encoders, preprocessing, chunked prompt eval | `mtmd_tokenize`, `mtmd_encode_chunk`, `mtmd_helper_eval_chunks`, `clip_image_batch_encode` | +| Audio generation | `../../llama.cpp/tools/mtmd/mtmd-helper-gen.cpp`, `../../llama.cpp/tools/mtmd/models/qwen3tts-gen.cpp`, `pockettts-gen.cpp` | TTS stage 2/3 graphs and stateful driver | `mtmd_gen_audio_process`, `qwen3tts_gen_audio_pipeline`, `pockettts_gen_audio_pipeline` | +| Server | `../../llama.cpp/tools/server/server-context.cpp`, `server-queue.cpp`, `server-http.cpp` | slots, continuous batching, HTTP/SSE | `server_slot`, `update_slots`, `server_queue`, `set_chunked_content_provider` | +| Conversion | `../../llama.cpp/convert_hf_to_gguf.py`, `../../llama.cpp/conversion/base.py`, `../../llama.cpp/gguf-py/gguf/` | HF -> GGUF | `ModelBase`, `TextModel`, `MmprojModel`, `GGUFWriter` | + +## 2. Device layer + +**Backend abstraction.** [Verified] Three-level object model in `../../llama.cpp/ggml/include/ggml-backend.h`: `ggml_backend_reg_t` (a backend library, enumerates devices) -> `ggml_backend_dev_t` (a physical device with `ggml_backend_dev_props{name, description, memory_free/total, type, device_id, caps}`) -> `ggml_backend_t` (a "stream"/execution context created by `ggml_backend_dev_init`). Buffer types (`ggml_backend_buffer_type_t`) allocate `ggml_backend_buffer_t`. Device types: `CPU`, `GPU`, `IGPU` (integrated, host memory), `ACCEL` (BLAS/AMX used together with CPU), `META` (tensor-parallel wrapper) (`../../llama.cpp/ggml/include/ggml-backend.h#L134`). + +**Device capability flags.** [Verified] `ggml_backend_dev_caps{async, host_buffer, buffer_from_host_ptr, events, mmap_support}` (`../../llama.cpp/ggml/include/ggml-backend.h#L148`). `llama_context` enables pipeline parallelism only if all non-CPU devices report `async` and `events` (`../../llama.cpp/src/llama-context.cpp#L426`). + +**Backends in tree and registration.** [Verified] `../../llama.cpp/ggml/src/ggml-backend-reg.cpp#L115` `struct ggml_backend_registry` registers at static-init time (guarded by `GGML_USE_*`): CUDA, Metal, SYCL, Vulkan, WebGPU, zDNN, VirtGPU frontend, OpenCL, ZenDNN, Hexagon, CANN, BLAS, RPC, OpenVINO, ET, and CPU last. HIP and MUSA compile the CUDA source tree (`ggml-hip/`, `ggml-musa/` are thin CMake wrappers; `ggml-hip` has 0 source lines). `ggml_backend_load_all_from_path` (`#L578`) dlopens `ggml-{blas,zendnn,cann,cuda,hip,metal,rpc,sycl,vulkan,virtgpu,opencl,hexagon,musa,openvino,cpu}` plugins when `GGML_BACKEND_DL` is used; `ggml_backend_load_best` picks the best CPU variant (`GGML_CPU_ALL_VARIANTS`). Backend order in the registry defines scheduler priority (GPU first, CPU last). + +**Device discovery/selection in llama.** [Verified] `../../llama.cpp/src/llama.cpp#L139-L303`: `llama_model_load_from_file_impl` enumerates `ggml_backend_dev_count()`; if `params.devices` is NULL it collects GPU devices, then IGPU devices only if no GPU, RPC servers first; `split_mode == LLAMA_SPLIT_MODE_NONE` keeps only `main_gpu`. `llama_model_params{devices, tensor_buft_overrides, n_gpu_layers, split_mode, load_mode, lazy_mode, main_gpu, tensor_split, use_extra_bufts, no_host, no_alloc}` (`../../llama.cpp/include/llama.h`). CLI `--device`, `--list-devices` (`../../llama.cpp/docs/build.md#L838`). + +**Weight placement.** [Verified] `../../llama.cpp/src/llama-model.cpp#L1031` `make_cpu_buft_list` builds an ordered candidate list: ACCEL device bufts, GPU host (pinned) bufts (unless `no_host`), CPU extra bufts (AMX/KleidiAI/repack, when `use_extra_bufts`), then plain CPU. `make_gpu_buft_list` (`#L1093`) adds split-row bufts for `LLAMA_SPLIT_MODE_ROW`, the device buft, and GPU extra bufts. Layers `[n_layer+1-n_gpu_layers, n_layer]` map to GPU devices proportionally to `tensor_split` (`#L1479-L1504`); input embeddings always on CPU (`dev_input`). Per-tensor overrides via regex `tensor_buft_overrides` (`--override-tensor`). Each weight is placed in the first buft in its layer list whose device `supports_op` for a probe op (`llama_model_loader` `select_weight_buft`; [Inferred] from function names at `#L1031-L1138` and loader code). + +**Memory allocation & pools.** [Verified] Weights: `ggml_backend_alloc_ctx_tensors_from_buft` (`../../llama.cpp/ggml/src/ggml-alloc.c#L1239`), or zero-copy `ggml_backend_dev_buffer_from_host_ptr` from the mmap'd file when the device supports it and the buft is the default one (`../../llama.cpp/src/llama-model.cpp#L1734`). `LLAMA_LOAD_MODE_{AUTO,MMAP,MLOCK,MMAP_MLOCK,NONE}` control mmap/mlock (`../../llama.cpp/src/llama-model-loader.cpp#L559`). Compute buffers: `ggml_gallocr` in `ggml-alloc.c` plans in-place reuse (`ggml_op_can_inplace`, `#L22`), splits large allocations into chunks (`struct vbuffer`, `#L398`), and reallocates only when `ggml_gallocr_needs_realloc` (`#L1009`). KV cache: one `ggml_context` per buffer type, K/V tensors allocated per layer (`../../llama.cpp/src/llama-kv-cache.cpp#L104-L152`); `offload_kqv` places the cache on the layer's device. Output logits/embeddings: pinned host buffer `buf_output` (`../../llama.cpp/src/llama-context.cpp#L380`). Backend-internal pools (e.g. CUDA VMM pool) are inside each backend ([Verified] `GGML_CUDA_NO_VMM` option in `../../llama.cpp/ggml/CMakeLists.txt#L204`). + +**Host <-> device transfers.** [Verified] `ggml_backend_tensor_set/get` (sync), `_async` variants, strided 2D variants `ggml_backend_tensor_set_2d_async` (`../../llama.cpp/ggml/include/ggml-backend.h#L86-L96`), `ggml_backend_tensor_copy_async` between backends with automatic sync fallback (`#L116`). The scheduler copies split inputs: user inputs (`GGML_TENSOR_FLAG_INPUT`) are copied synchronously after waiting on the copy's event; other inputs use `cpy_tensor_async` if the backend supports it, else sync copy (`../../llama.cpp/ggml/src/ggml-backend.cpp#L1670-L1760`). MoE weights that live on host are copied expert-by-expert only for the experts used (`#L1690`). Model loading uploads via `load_all_data` with optional async upload backend ([Inferred] from `upload_backend`/`ggml_backend_event` usage grep in `llama-model-loader.cpp`). + +**Synchronization / streams.** [Verified] One `ggml_backend_t` per device = one stream. `ggml_backend_synchronize`, `ggml_backend_event_{new,record,wait,synchronize}` (`ggml-backend.h#L121-L128`). Scheduler keeps `events[GGML_SCHED_MAX_BACKENDS=16][GGML_SCHED_MAX_COPIES=4]` (`ggml-backend.cpp#L763-L772, #L819`) and rotates `cur_copy` per graph for pipeline parallelism; `n_copies = parallel ? 4 : 1` (`#L1865`). `llama_context::graph_compute` calls `ggml_backend_sched_graph_compute_async` and results are fetched with async `ggml_backend_tensor_get_async` then `llama_synchronize` (`../../llama.cpp/src/llama-context.cpp#L2492`, `#L714`). + +**Tensor parallelism / multi-device.** [Verified] `GGML_BACKEND_DEVICE_TYPE_META` + `ggml_backend_meta_device` (`../../llama.cpp/ggml/src/ggml-backend-meta.cpp`) implement experimental tensor parallelism (`LLAMA_SPLIT_MODE_TENSOR`, `../../llama.cpp/docs/multi-gpu.md#L80`); NCCL optional (`GGML_CUDA_NCCL`). Default multi-GPU is layer split with pipeline parallelism. + +**RPC backend.** [Verified] `../../llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp#L61` defines `rpc_cmd` {ALLOC_BUFFER, GET_ALIGNMENT, GET_MAX_SIZE, BUFFER_GET_BASE, FREE_BUFFER, BUFFER_CLEAR, SET_TENSOR, SET_TENSOR_HASH, GET_TENSOR, COPY_TENSOR, GRAPH_COMPUTE, GET_DEVICE_MEMORY, INIT_TENSOR, GET_ALLOC_SIZE, HELLO, DEVICE_COUNT, GRAPH_RECOMPUTE, MEMSET_TENSOR}; graphs are serialized and computed remotely; `SET_TENSOR_HASH` dedups uploads > 10 MiB. Server binary: `../../llama.cpp/tools/rpc/rpc-server.cpp`. Apple transport variant `transport-apple.cpp`. + +**Portability (OS/arch actually in tree).** [Verified] +- CPU arches: `ggml/src/ggml-cpu/arch/{arm,x86,riscv,loongarch,powerpc,s390,wasm}`; build options for SSE4.2/AVX/AVX2/AVX512(+VNNI/BF16)/AMX, RVV/Zfh/Zvfh, LASX/LSX, VXE (`../../llama.cpp/ggml/CMakeLists.txt#L154-L181`). Android NDK arm64-v8a documented (`../../llama.cpp/docs/android.md`), Termux build, WASM/Emscripten (`LLAMA_WASM_*`). +- GPU/NPU: CUDA (arch list incl. `86-real`, `89-real`, `120a/121a-real`; [Inferred] Jetson Orin sm_87 would rely on `native` or user-provided `CMAKE_CUDA_ARCHITECTURES`, `../../llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt#L28-L54`), HIP, MUSA, Metal (macOS/iOS/visionOS/tvOS via `build-xcframework.sh`), Vulkan (Windows/Linux/Android/macOS via MoltenVK or KosmicKrisp, `../../llama.cpp/docs/build.md#L402-L557`), OpenCL for Adreno (Android, Windows Arm64; Adreno 750/810/830/840/X1-85/X2-90 listed, `../../llama.cpp/docs/backend/OPENCL.md#L47-L54`), Hexagon HTP v73+ via Hexagon SDK containers (`../../llama.cpp/docs/backend/snapdragon/README.md`), CANN, SYCL (Intel GPU), OpenVINO, zDNN (IBM Z), WebGPU (Dawn; browser via Emscripten), VirtGPU (VM API remoting), ET (RISC-V ET-SOC). + +## 3. Kernel layer + +**Operator set.** [Verified] `enum ggml_op` in `../../llama.cpp/ggml/include/ggml.h#L482-L593` has 102 entries incl. `MUL_MAT`, `MUL_MAT_ID` (MoE), `FLASH_ATTN_EXT`, `ROPE`, `RMS_NORM`, `GROUP_NORM`, `L2_NORM`, `SSM_CONV/SSM_SCAN`, `RWKV_WKV6/7`, `GATED_LINEAR_ATTN`, `GATED_DELTA_NET`, `LIGHTNING_INDEXER`, `DSV4_HC_*`, `SOLVE_TRI`, `TRI`, conv family (`CONV_2D`, `CONV_3D`, `CONV_2D_DW`, `CONV_TRANSPOSE_1D/2D`, `IM2COL`, `IM2COL_3D`, `COL2IM_1D`), `POOL_1D/2D`, `UPSCALE`, `PAD`, `PAD_REFLECT_1D`, `ROLL`, `ARANGE`, `TIMESTEP_EMBEDDING`, `ARGSORT`, `TOP_K`, `CUMSUM`, `SET_ROWS`, `GET_ROWS`, optimizer ops (`OPT_STEP_ADAMW/SGD`, `CROSS_ENTROPY_LOSS`), `MAP_CUSTOM1..3`, `CUSTOM`, `UNARY`, `GLU`. Unary and GLU sub-ops are separate enums (`#L596`, `#L623`). Per-backend support matrix is generated into `../../llama.cpp/docs/ops.md` from `docs/ops/*.csv` (columns: BLAS, CANN, CPU, CUDA, ET, HTP, MTL, OpenCL, SYCL, Vulkan, WebGPU, ZenDNN, zDNN) via `test-backend-ops support --output csv` + `../../llama.cpp/scripts/create_ops_docs.py`. + +**Dispatch mechanism.** [Verified] There is no per-op registry table across backends. Each backend implements `ggml_backend_device_i::supports_op(dev, op)` and `offload_op(dev, op)` (`../../llama.cpp/ggml/src/ggml-backend-impl.h#L205-L212`) plus a `graph_compute(backend, cgraph)` entry (`#L146`) that walks the nodes and switches on `op->op`. The scheduler queries `supports_op`/`supports_buft` to assign nodes (Section 5b). Examples: `ggml_backend_cpu_device_supports_op` (`../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp#L424`, which also delegates to extra buffer types' `supports_op`), `ggml_backend_cuda_device_supports_op` (`../../llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu#L5060`), `ggml_backend_metal_device_supports_op` (`../../llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp#L741`), `ggml_backend_vk_device_supports_op` (`../../llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp#L19037`), `ggml_opencl_supports_op` (`../../llama.cpp/ggml/src/ggml-opencl/ggml-opencl.cpp#L8393`), `ggml_backend_hexagon_device_supports_op` (`../../llama.cpp/ggml/src/ggml-hexagon/ggml-hexagon.cpp#L5831`; HTP op list at `#L4962-L4992`: FA, MUL_MAT(_ID), binary ops, CPY, GET/SET_ROWS, ARGSORT, norms, SOFT_MAX, SSM_CONV, GATED_DELTA_NET, ROPE, ...). `offload_op` lets a GPU claim ops on host-resident weights when the batch is large enough (CUDA: `GGML_OP_OFFLOAD_MIN_BATCH`, default 32, `ggml-cuda.cu#L5714`). + +**CPU kernel selection.** [Verified] `type_traits_cpu[GGML_TYPE_COUNT]` (`../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c#L215`) maps each quant type to `{from_float, vec_dot, vec_dot_type, nrows}`. In `ggml_compute_forward_mul_mat` (`../../llama.cpp/ggml/src/ggml-cpu/ops.cpp#L5240-L5290`) the activation matrix is quantized to `vec_dot_type` (e.g. Q8_0/Q8_K) into `params->wdata`, then `vec_dot` runs over quantized weight blocks; there is no full dequantization of weights. Optional tinyBLAS `llamafile_sgemm` (`../../llama.cpp/ggml/src/ggml-cpu/llamafile/sgemm.cpp`, `GGML_LLAMAFILE`) and BLAS/Accelerate for large F32/F16 matmuls. + +**CPU extra buffer types (repack / KleidiAI / AMX).** [Verified] `ggml_backend_cpu_get_extra_buffer_types` (`../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp#L42`) registers AMX, SpacemiT RISC-V, KleidiAI, and repack buffer types. An extra buffer type implements `extra_buffer_type::supports_op` and `get_tensor_traits` (`../../llama.cpp/ggml/src/ggml-cpu/traits.h#L29-L31`); `tensor_traits::compute_forward` overrides the kernel for ops whose weight lives in that buffer. `repack.cpp` converts Q4_0 -> Q4_0x4/x8/x16, Q4_K -> Q4_Kx8/x16, Q2_K/Q5_K/Q6_K, Q8_0, IQ4_NL, MXFP4 into interleaved blocks at load time (`make_block_*`, `repack_*_bl`, `ggml_repack_get_optimal_repack_type` at `#L4528`). KleidiAI (`../../llama.cpp/ggml/src/ggml-cpu/kleidiai/kleidiai.cpp`) selects Arm microkernels by dotprod/i8mm/SVE/SME2 at runtime (`../../llama.cpp/docs/build.md#L616-L712`). + +**Quantization formats.** [Verified] `enum ggml_type` (`../../llama.cpp/ggml/include/ggml.h#L390-L433`): F32, F16, BF16, F64, I8/I16/I32/I64, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1, Q2_K..Q6_K, Q8_K, IQ2_XXS/XS/S, IQ3_XXS/S, IQ1_S/M, IQ4_NL/XS, TQ1_0/TQ2_0, MXFP4, NVFP4, Q1_0, Q2_0. Quantize/dequantize reference in `../../llama.cpp/ggml/src/ggml-quants.c`; model-level `llama_ftype` (42 `LLAMA_FTYPE_MOSTLY_*` entries) and `llama_model_quantize` in `../../llama.cpp/src/llama-quant.cpp` (imatrix-aware, per-tensor type overrides, layer pruning). GPU: CUDA `mmq.cuh` (int8 tensor-core MMQ on quantized weights), `mmvq.cu` (matvec), `dequantize.cuh`/`convert.cu`; Metal kernels in `../../llama.cpp/ggml/src/ggml-metal/kernels/` (23 files incl. `dequantize.h`); Vulkan 180 shader sources in `../../llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/`; OpenCL 178 `.cl` kernels (`../../llama.cpp/ggml/src/ggml-opencl/kernels/`), Adreno path supports Q4_0/Q4_1/Q4_K/Q6_K/MXFP4 ([Verified] `../../llama.cpp/docs/backend/OPENCL.md#L59-L83`). KV cache may be quantized (`type_k/type_v`, e.g. q8_0), Flash-attention kernels handle quantized KV on CPU/CUDA/Metal/Vulkan ([Inferred] from `GGML_CUDA_FA_ALL_QUANTS` and `k_vec_dot_type` in `ops.cpp#L8688`). + +**Compilation / JIT / graph capture.** [Verified] +- Metal: shaders compiled from embedded source at runtime with `newLibraryWithSource` per op-source, or loaded from a prebuilt `.metallib` (`GGML_METAL_EMBED_LIBRARY`, `../../llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m#L300-L395`); pipelines cached (`ggml_metal_pipelines_t`). Metal 4 tensor API gated by `has_tensor`. +- Vulkan: GLSL compiled to SPIR-V at build time by the host tool `vulkan-shaders-gen` (`../../llama.cpp/ggml/src/ggml-vulkan/CMakeLists.txt#L174-L199`, `../../llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp`); pipelines created lazily at runtime; cooperative-matrix variants selected by device features (`coopmat_support`, `ggml-vulkan.cpp#L890`). +- OpenCL: kernels embedded (`GGML_OPENCL_EMBED_KERNELS`) and built at init with an on-disk program binary cache (`../../llama.cpp/ggml/src/ggml-opencl/cl-program-cache.cpp`); optional Adreno prebuilt binary kernel library (`GGML_OPENCL_USE_ADRENO_BIN_KERNELS`). +- CUDA: CUDA graph capture for decode graphs (`GGML_CUDA_GRAPHS`, `ggml_cuda_graph_evaluate_and_capture` at `../../llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu#L4181`; keyed per `graph_key`); HIP graphs, SYCL graphs, MUSA graphs (experimental) also have options (`../../llama.cpp/ggml/CMakeLists.txt#L207-L250`). +- Hexagon: HTP kernels compiled with the Hexagon SDK into `libggml-htp-v73.so`-style DSP libraries invoked over FastRPC (`../../llama.cpp/ggml/src/ggml-hexagon/htp/`, `htp_iface.idl`; [Verified] build log excerpt in `../../llama.cpp/docs/backend/snapdragon/README.md`). +- Backend-side graph rewriting hook: `ggml_backend_i::graph_optimize` (`../../llama.cpp/ggml/src/ggml-backend-impl.h#L155`) is invoked per split (`ggml-backend.cpp#L1462`). llama also probes fused ops per backend (`llm_fused_op_probe`, `llama_context::resolve_fused_ops`, `../../llama.cpp/src/llama-context.cpp#L35`, `#L505`). + +**Fallback when an op is unsupported.** [Verified] In `ggml_backend_sched_backend_id_from_cur` (`../../llama.cpp/ggml/src/ggml-backend.cpp#L921`): a node whose weight buffer's backend does not `supports_op` searches other backends that support both the buffer type and op; if none, the weight is copied ("the weight will need to be copied"). Graph inputs default to the last backend (CPU). Pass 2/3 expand assignments to neighbours and "upgrade" nodes to higher-priority backends only when buffer types are compatible; remaining unsupported nodes end up on CPU, producing extra splits and copies (`#L1124-L1297`). `docs/ops.md` marks partial support; the ET backend doc explicitly states "some/most operations will likely fallback to CPU backend" (`../../llama.cpp/docs/backend/ET.md`). + +**Extension points (custom op / backend).** [Verified] Custom ops: `ggml_map_custom1/2/3` and `ggml_custom_4d(ctx, type, ne0..3, args, n_args, fun, n_tasks, userdata)` with `ggml_custom_op_t` executed by the CPU backend (`../../llama.cpp/ggml/include/ggml.h#L2679-L2747`). New backend: implement `ggml_backend_reg_i`, `ggml_backend_device_i`, `ggml_backend_i`, `ggml_backend_buffer_type_i`, `ggml_backend_buffer_i` vtables (`ggml-backend-impl.h`), export with `GGML_BACKEND_DL_IMPL(reg_fn)` (`#L258-L285`), add to `ggml-backend-reg.cpp` and CMake. Optional proc-address extensions: `ggml_backend_set_n_threads`, `ggml_backend_dev_get_extra_bufts`, `ggml_backend_set_abort_callback`, `ggml_backend_get_features` (`ggml-backend.h#L205-L225`). New model arch: `../../llama.cpp/docs/development/HOWTO-add-model.md` (conversion class in `conversion/`, arch enum in `llama-arch.cpp`, `llama_model_` class in `src/models/models.h` with `load_arch_hparams/load_arch_tensors/build_graph`). Tests for new ops go in `../../llama.cpp/tests/test-backend-ops.cpp`. + +## 4. Model runner + +**Formats and conversion.** [Verified] Runtime format is GGUF only (`../../llama.cpp/ggml/src/gguf.cpp`, `../../llama.cpp/gguf-py/gguf/gguf_writer.py`, `gguf_reader.py`, `constants.py`, `tensor_mapping.py`). `../../llama.cpp/convert_hf_to_gguf.py` (312 lines) dispatches to registered classes in `../../llama.cpp/conversion/` (`ModelBase`, `TextModel`, `MmprojModel` in `../../llama.cpp/conversion/base.py`; 100+ per-family files incl. `qwen3tts.py`, `pockettts.py`, `wavtokenizer.py`, `talkie.py`, `ultravox.py`, `qwen3vl.py`). Multimodal encoders are converted into a separate `mmproj` GGUF (`--mmproj`). Also `convert_lora_to_gguf.py`, `convert_llama_ggml_to_gguf.py`, `llama-quantize` (`../../llama.cpp/tools/quantize/`), `llama-imatrix`, `gguf-split`. Model sharding/splits supported (`llama_model_load_from_splits`, `llama_get_list_splits` in `llama-model-loader.cpp#L82`). Models can also be constructed programmatically without a file: `llama_model_init_from_user` (`../../llama.cpp/src/llama.cpp#L446`). + +**Load pipeline.** [Verified] `llama_model_load_from_file_impl` (`../../llama.cpp/src/llama.cpp#L380`) -> device list -> `llama_model_load` (`#L316`) -> `llama_model_loader` parses GGUF metadata (+ splits), `model.load_arch/hparams/vocab/stats`, `model.load_tensors` (buffer placement, mmap zero-copy or `load_all_data` upload; `../../llama.cpp/src/llama-model.cpp#L1403-L1834`), optional `check_tensors`, progress callback that can abort loading. `lazy_mode` allows on-demand reading of arch-marked tensors. 152 `LLM_ARCH_*` enums (`../../llama.cpp/src/llama-arch.h`). + +**Execution lifecycle.** [Verified] +1. `llama_backend_init()`; optionally `ggml_backend_load_all()` for DL builds (`../../llama.cpp/examples/simple/simple.cpp#L82`). +2. `llama_init_from_model(model, params)` -> `llama_context` ctor (`../../llama.cpp/src/llama-context.cpp#L83`): computes `n_ctx` padded to 256 and `n_ctx_seq = n_ctx / n_seq_max` unless `kv_unified` (`#L288-L301`), `create_memory`, inits one `ggml_backend_t` per model device + ACCEL devices + CPU (`#L331-L357`), collects `set_n_threads` hooks, allocates output buffer, decides pipeline parallelism, then `sched_reserve()` (`#L582`). +3. `sched_reserve`: builds `ggml_backend_sched_new(backends, bufts, n, max_nodes, pipeline_parallel, op_offload)` and reserves compute buffers by building worst-case graphs for prompt-processing (`n_ubatch` tokens) and token-generation (1 token) shapes via `graph_reserve` (`#L2416`), logging split counts `n_splits_pp/tg` and buffer sizes. If allocation fails with pipeline parallelism it retries without (`#L637`). +4. Warmup: `llama_set_warmup(ctx, true)` and a decode of BOS/EOS is done by `common` init (deprecated-marked in API but present). +5. Step: `llama_decode(ctx, batch)` (`#L1644`) -> `balloc->init` validates the `llama_batch` (tokens or embeddings, positions, `seq_id`, `logits` output flags) -> `memory_update(false)` applies pending shifts/copies -> `memory->init_batch(balloc, n_ubatch, output_all)` splits into ubatches and finds KV slots (retrying once after `memory_update(true)` optimization) -> `output_reserve` -> for each ubatch `process_ubatch` (`#L1334`): `mctx->apply()`, graph reuse check, else `ggml_backend_sched_reset` + `model.build_graph(gparams)` + `ggml_backend_sched_alloc_graph`; `res->set_inputs(&ubatch)`; `graph_compute` (async) -> logits/embeddings copied with `ggml_backend_tensor_get_async` into host buffers, reordered by `output_reorder`. `llama_encode` handles encoder-only / encoder half of enc-dec (T5) and non-causal models (`#L1406`). +6. `llama_get_logits_ith`, `llama_get_embeddings_{ith,seq}`, backend-sampler results `llama_get_sampled_token_ith` (`../../llama.cpp/include/llama.h#L1035-L1092`). + +**Dynamic shapes / graph reuse.** [Verified] Graphs are built per ubatch as `ggml_cgraph` (no static shape compilation). `llm_graph_params` records `ubatch` shape, `n_outputs`, memory context, and each `llm_graph_input_*::can_reuse` compares against the previous graph (`../../llama.cpp/src/llama-graph.h#L100-L760`, `#L771-L892`); when identical topology, `process_ubatch` skips rebuilding and reallocating (`n_reused++`), which is what makes CUDA graph capture effective. Env `LLAMA_GRAPH_REUSE_DISABLE` disables it (`../../llama.cpp/src/llama-context.cpp#L280`). Because `ggml_gallocr` reserved worst-case buffers, varying `n_tokens <= n_ubatch` needs no reallocation (`ggml_gallocr_needs_realloc`). + +**Batch splitting.** [Verified] `llama_batch_allocr::split_simple(n_ubatch)` (sequential chunk), `split_equal(n_ubatch, sequential, n_keep_tail)` (equal-length per-sequence groups, used by non-unified KV and recurrent memories), `split_seq(n_ubatch)` (one sequence per ubatch) (`../../llama.cpp/src/llama-batch.h#L103-L111`). KV cache picks `split_simple` when `n_stream == 1`, else `split_equal` (`../../llama.cpp/src/llama-kv-cache.cpp#L713`). `llama_ubatch` holds `token/embd/pos/n_seq_id/seq_id/seq_id_unq/output` (`llama-batch.h#L15`). + +**State and KV cache.** [Verified] +- Interface: `llama_memory_i` {`init_batch`, `init_full`, `init_update`, `clear`, `seq_rm/seq_cp/seq_keep/seq_add/seq_div`, `seq_pos_min/max`, `state_write/read`, `memory_breakdown`} and `llama_memory_context_i` {`next`, `apply`, `get_ubatch`, `get_status`} (`../../llama.cpp/src/llama-memory.h#L51-L126`). Public C API `llama_memory_*` (`../../llama.cpp/include/llama.h#L739-L805`). +- Factory: `llama_model::create_memory` (`../../llama.cpp/src/llama-model.cpp#L2232-L2700`) returns `nullptr` for encoder-only/diffusion/WavTokenizer archs (no cache), `llama_kv_cache_msa` (MiniMax-M3), `llama_kv_cache` (standard), `llama_kv_cache_dsa` / `dsa_iswa` (DeepSeek sparse attention), `llama_kv_cache_iswa` (two caches: full + sliding-window, `../../llama.cpp/src/llama-kv-cache-iswa.h`), `llama_kv_cache_dsv4`, `llama_memory_recurrent` (Mamba/RWKV: per-seq `r_l`/`s_l` state tensors, `n_rs_seq` rollback snapshots, `../../llama.cpp/src/llama-memory-recurrent.h#L70-L113`), `llama_memory_hybrid`, `hybrid_iswa`, `hybrid_idx`. +- Layout: `llama_kv_cache` stores per-layer `k_l/v_l` tensors of `kv_size` cells (K contiguous per head, V optionally transposed `v_trans` for non-FA path), one **stream** per sequence when `unified=false` (`n_stream = n_seq_max`) or a single shared stream when unified (`../../llama.cpp/src/llama-kv-cache.cpp#L84`, `#L135-L152`). Cells are tracked by `llama_kv_cells` with `pos`, `shift`, `seq` bitset (`std::bitset`) and a `seq_pos` index (`../../llama.cpp/src/llama-kv-cells.h#L489-L524`). `find_slot(ubatch, cont)` (`llama-kv-cache.cpp#L898`) scans the ring buffer for free cells (contiguous or not; non-contiguous slots are expressed via `k_idxs/v_idxs` and `GGML_OP_SET_ROWS`), honoring SWA masking to free expired cells. There is **no paging** and no defragmentation step any more (`defrag_thold` marked deprecated in `llama_context_params`); eviction is explicit via `llama_memory_seq_rm`/context-shift (`seq_add` with K-shift graph, `update(do_shift)` at `#L817`). +- Sharing across requests: `seq_cp` copies cells between sequences (full-buffer copy only for non-unified streams, `#L496-L506`); server prompt cache saves idle slot KV to RAM and restores by common prefix; `llama_state_seq_{get,set}_data(_ext)` with flags `SWA_ONLY`, `PARTIAL_ONLY`, `ON_DEVICE` (`../../llama.cpp/include/llama.h#L906-L916`), file variants `llama_state_seq_save_file/load_file`, whole-context `llama_state_get_data/set_data`. +- KV dtypes: `type_k/type_v` (F16 default, quantized allowed), `offload_kqv`, `flash_attn_type`, `swa_full`, `kv_unified` in `llama_context_params`. + +**Multimodal stages.** [Verified] Encoders are in `libmtmd` (`../../llama.cpp/tools/mtmd/`), not in `libllama`: `mtmd_init_from_file(mmproj, model, params)` loads a `clip_ctx` per modality (vision and/or audio, with own `ggml_backend_sched` and a chosen device `mtmd_context_params.device`, `../../llama.cpp/tools/mtmd/clip.cpp#L153-L202`). Pipeline (`../../llama.cpp/tools/mtmd/README-dev.md#L18`): bitmap (RGB or PCM) -> `mtmd_tokenize` (splits prompt into text/image/audio chunks, runs preprocessor: `mtmd_image_preprocessor_*` slicing/resizing in `mtmd-image.cpp`, mel spectrogram in `mtmd-audio.cpp`) -> `mtmd_encode_chunk`/`mtmd_batch_encode` (runs `clip_image_batch_encode`, graph built by `clip_get_graph_builder` per projector type at `clip.cpp#L930`, computed at `#L4437-L5749`) -> `mtmd_get_output_embd` -> `mtmd_helper_decode_image_chunk` feeds embeddings to `llama_decode` via `llama_batch.embd` (with M-RoPE 2D positions via `mtmd_decoder_pos`, non-causal attention toggled by `llama_set_causal_attn` for some models) (`../../llama.cpp/tools/mtmd/mtmd-helper.cpp#L100-L293`). Decoders/vocoders for TTS live in mtmd as "gen" graphs (Section 9). Diffusion LMs are driven by `../../llama.cpp/examples/diffusion/diffusion.cpp` on top of `llama_decode` with `llama_model_is_diffusion`. + +## 5. Scheduler + +### 5a. Request / token / pipeline-level scheduling + +**Library level (libllama).** [Verified] `libllama` has no request scheduler: the caller composes a `llama_batch` with tokens from any number of sequences (`seq_id` per token, up to `n_seq_max`, `LLAMA_MAX_SEQ=256`), and `llama_decode` processes it as ubatches of at most `n_ubatch` tokens, sharing weights across sequences (this is the batching primitive). `examples/batched` and `examples/parallel` demonstrate multi-sequence decoding with `llama_memory_seq_cp` for shared prefixes (`../../llama.cpp/examples/parallel/parallel.cpp#L263-L307`). Cancellation inside a decode: `llama_set_abort_callback` (CPU backend `ggml_cplan.abort_callback` checked per node, `../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c#L3150`; Metal also supports an abort callback, `../../llama.cpp/ggml/src/ggml-metal/ggml-metal-context.m#L620`); other backends: [Unknown]/not observed. Backpressure: `llama_decode` returns `1` when no KV slot is found (`../../llama.cpp/src/llama-context.cpp#L1779`), leaving the caller to shrink or retry. + +**Server level (tools/server).** [Verified] +- Slots: `n_parallel` `server_slot`s, each bound to a `seq_id`, with `n_ctx_slot = n_ctx_seq` (or capped with `kv_unified_per_slot`) (`../../llama.cpp/tools/server/server-context.cpp#L239`, `#L1250-L1295`). States: `IDLE, WAIT_OTHER, STARTED, PROCESSING_PROMPT, DONE_PROMPT, GENERATING` (`#L100`). +- Queue: `server_queue` (`../../llama.cpp/tools/server/server-queue.cpp`) is a mutex/condvar-protected deque; `post()`, `defer()` (task waits for a free slot), `pop_deferred_task(id_slot)`, `start_loop` runs `process_single_task` then `update_slots` on one worker thread; `yield_to_queue` lets `update_slots` interleave new tasks (`#L223`). +- Continuous batching: `update_slots()` (`#L2777`) adds one sampled token per generating slot (plus speculative draft tokens) to a shared `llama_batch`, then fills remaining capacity up to `n_batch` with pending prompt tokens (`#L3087-L3129`), reusing common prefixes (`get_common_prefix`, `n_cache_reuse` KV shifting `#L3202-L3268`, context checkpoints), decodes in chunks of `n_batch` (`#L2853-L2873`; on KV-slot failure it halves `n_batch` and retries), and dispatches results per slot. Slots that cannot batch together (`can_batch_with`, e.g. different LoRA) are processed in separate iterations (`#L2973`, `#L3108`). +- Prefill vs decode: no separate phases/engines; prompt tokens and generation tokens share ubatches (chunked prefill is natural since prompts are consumed `n_batch` at a time). Context shift when full (`n_discard`, `seq_rm` + `seq_add`, `#L2898-L2948`), disabled for multimodal. +- Cancellation: `SERVER_TASK_TYPE_CANCEL` releases the slot (`#L2438`); `post()` drops pending tasks with the same id (`server-queue.cpp#L31`). HTTP disconnect during streaming ends the chunked provider. +- Multi-stage: speculative decoding (draft model, EAGLE-3, DFlash, DSpark, n-gram variants; `../../llama.cpp/docs/speculative.md`, `../../llama.cpp/common/speculative.h`) runs draft + verify inside a slot; parent/child slots exist for `n > 1` completions (`launch_slots_with_parent_task`, `#L2264`). Router mode (`--models-dir`, `server-models.cpp`) spawns/unloads child `llama-server` processes per model and proxies requests (`../../llama.cpp/tools/server/README.md#L230-L233`, `#L1800-L2040`). +- Backpressure: `n_parallel` bounds concurrency; excess tasks are deferred; `/slots` and `/metrics` expose state; `--cache-ram` bounds prompt-cache RAM. + +**TTS pipeline scheduling.** [Verified] `mtmd_helper_gen_audio` (`../../llama.cpp/tools/mtmd/mtmd-helper-gen.cpp`) is a single-sequence stateful driver: `set_input` -> `step_prompt(n_batch)` loops -> per-frame `step_gen(sampled, h_state)` running GEN_CODE then buffered GEN_WAV flushes (`flush_gen_wav`, window of 72 frames for Qwen3-TTS `window_frames`, `#L400-L437`) -> `get_output`. No batching across TTS requests; the server does not expose it. + +### 5b. Graph / operator / thread-level scheduling + +**Graph split scheduling (`ggml_backend_sched`).** [Verified] `ggml_backend_sched_split_graph` (`../../llama.cpp/ggml/src/ggml-backend.cpp#L1066`) runs five passes: (1) assign nodes/leafs with pre-allocated tensors (weights -> their backend; graph inputs -> CPU; `op_offload` may move host-weight ops to a GPU if `offload_op` says so, `#L921-L995`); (2) expand assignments to adjacent nodes, GPU backends up/down but never expanding CPU (`#L1124`); (3) upgrade nodes to higher-priority backends when buffer types are compatible (`#L1204`); (4) assign remaining sources from destination/view_src (`#L1265`); (5) cut the node list into contiguous same-backend splits, record split inputs that need copies (`#L1297-L1423`), call each backend's `graph_optimize` (`#L1462`), and build a copy graph with copy nodes inserted (`#L1475-L1513`). `ggml_backend_sched_alloc_splits` allocates via `ggml_gallocr` with per-node buffer ids, reallocating only if needed. `ggml_backend_sched_compute_splits` (`#L1643`) executes splits in order, one async `graph_compute` per split, with event-based cross-backend synchronization and `cur_copy` rotation; `callback_eval` can observe intermediate tensors and abort. Op ordering within a split is the topological order of `ggml_build_forward_expand`; there is no reordering/fusion in the scheduler itself (fusion is backend-internal, e.g. CUDA/Metal fusing RMS_NORM+MUL, probed via `llm_fused_op_probe`). + +**Threads (CPU backend).** [Verified] `ggml_threadpool` (`../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c#L481`) with `n_threads`, per-thread `cpumask`, priority (`GGML_SCHED_PRIO_{LOW,NORMAL,MEDIUM,HIGH,REALTIME}` mapped to SCHED_FIFO/OTHER on Linux/macOS, thread priority classes on Windows, `#L2557-L2700`), polling level, pause/resume (`ggml_threadpool_pause/resume`, `../../llama.cpp/ggml/include/ggml-cpu.h#L58-L62`), OpenMP mode when `GGML_OPENMP` (default ON; Android docs recommend OFF). Each op is split across threads by row ranges inside `ggml_compute_forward_*`, with `ggml_barrier` between ops (`#L576`). `llama_context` keeps separate `threadpool` and `threadpool_batch` (`n_threads` vs `n_threads_batch`) and sets them per compute (`../../llama.cpp/src/llama-context.cpp#L2492-L2510`); CLI exposes `--threads`, `--cpu-mask`, `--cpu-range`, `--prio`, `--poll` (`../../llama.cpp/tools/cli/README.md`). GPU backends use a single stream per device; Hexagon HTP has its own DSP worker pool (`htp/worker-pool.c` referenced in the Snapdragon doc build log). + +**Stream assignment.** [Verified] One stream per backend; pipeline parallelism overlaps up to 4 ubatches across devices via copies/events. Backends may additionally use internal multi-streams ([Unknown] not audited per backend). + +## 6. I/O layer + +**Text tokenization.** [Verified] Built-in, no external library: `llama_vocab_type` {NONE, SPM, BPE, WPM, UGM, RWKV, PLAMO2} (`../../llama.cpp/include/llama.h#L73-L79`), implementations `llm_tokenizer_spm_session`, `llm_tokenizer_bpe_session` (with `unicode_regex_split` pre-tokenizer regexes per `LLAMA_VOCAB_PRE_TYPE_*`), `llm_tokenizer_wpm_session`, `llm_tokenizer_ugm_session` (naive trie + Viterbi-style), `llm_tokenizer_rwkv_session`, `llm_tokenizer_plamo2_session`, plus `hybriddna` and `whitespace` variants (`../../llama.cpp/src/llama-vocab.cpp#L110-L1723`). Unicode tables in `../../llama.cpp/src/unicode-data.cpp`. API: `llama_tokenize`, `llama_token_to_piece`, `llama_detokenize` (`../../llama.cpp/include/llama.h#L1172-L1200`), special tokens (BOS/EOS/EOT/SEP/PAD/MASK/FIM), `llama_vocab_is_eog`. Chat templates: in-tree Jinja engine (`../../llama.cpp/common/jinja/README.md`, not minja) used by `common/chat.cpp` with tool-call grammar constraining and PEG-based output parsers (`../../llama.cpp/docs/autoparser.md`, `../../llama.cpp/common/peg-parser.cpp`); `llama_chat_apply_template` C API with built-in templates. Grammar-constrained sampling: GBNF (`../../llama.cpp/grammars/README.md`, `../../llama.cpp/src/llama-grammar.cpp`), JSON-schema -> grammar, optional LLGuidance. + +**Image preprocessing.** [Verified] Decoding via vendored `stb_image` (JPEG/PNG/...), WebP and video via `ffmpeg`/`ffprobe` subprocess (`MTMD_VIDEO` compile option, `../../llama.cpp/tools/mtmd/mtmd-helper.cpp#L36`, `#L419`, `../../llama.cpp/tools/mtmd/mtmd-helper.h#L23-L53`). Resizing with Pillow-compatible bilinear/bicubic (`img_tool::resize_pillow`, `../../llama.cpp/tools/mtmd/mtmd-image.cpp#L204`), smart-resize to patch multiples, padding, LLaVA-UHD/MiniCPM-V slicing (`mtmd_image_preprocessor_llava_uhd`, `#L485`), normalization with `image_mean/std` from the mmproj GGUF. + +**Audio preprocessing.** [Verified] Decoding via vendored `miniaudio` (WAV/MP3/FLAC, resampled to model sample rate, `mtmd-helper.cpp#L23-L33`). Mel spectrogram from a whisper.cpp-derived implementation: Hann window, Cooley-Tukey FFT, mel filterbank matrix (`mtmd_audio_cache::fill_mel_filterbank_matrix`, `fft_impl`, `../../llama.cpp/tools/mtmd/mtmd-audio.cpp#L13-L268`). `mtmd_get_audio_sample_rate` reports e.g. 16000 for Whisper-type encoders. Audio encoders in mtmd: `whisper-enc.cpp`, `conformer.cpp`, `parakeet.cpp`, `granite-speech.cpp`, `qwen3a.cpp`, `mimo-audio.cpp`, `gemma4a.cpp/gemma4ua.cpp` (`../../llama.cpp/tools/mtmd/models/`). + +**Streaming.** [Verified] Token streaming: server SSE via `httplib` chunked content provider (`../../llama.cpp/tools/server/server-http.cpp#L546-L576`), `server-stream.cpp` assembles OpenAI/Anthropic-style deltas; CLI prints pieces as sampled. Audio streaming output: `llama-tts` accumulates PCM and writes one WAV (`write_wav16`, `../../llama.cpp/tools/tts/tts.cpp#L53`); the helper API is frame-stepped (`step_gen` returns after each frame; GEN_WAV state carried via `state_data` so windows can be decoded incrementally, `mtmd.h#L426-L433`), so chunked audio streaming is achievable at the application layer [Inferred]. No audio-streaming HTTP endpoint exists. + +**Buffering/postprocessing.** [Verified] Logits are returned only for tokens flagged in `llama_batch.logits`; output rows are reordered to batch order (`output_reorder`). Sampling: CPU sampler chain (greedy, dist, top-k/p, min-p, typical, temp, XTC, top-n-sigma, mirostat, penalties, DRY, adaptive-p, logit-bias, grammar, infill; `../../llama.cpp/include/llama.h#L1367-L1524`) or experimental backend samplers executed in-graph (`llama_set_sampler`, `llama_get_sampled_token_ith`; `llm_graph_input_sampling` in `llama-graph.h#L744`). Detokenization is incremental via `llama_token_to_piece`; server handles partial UTF-8. Vocoder/codec postprocessing for TTS is inside mtmd gen graphs (Section 9). + +**Application boundaries.** [Verified] +- C API: `../../llama.cpp/include/llama.h` (`llama_*`), `../../llama.cpp/tools/mtmd/mtmd.h` + `mtmd-helper.h` (`mtmd_*`), ggml headers. +- C++ helpers: `../../llama.cpp/common/` (`common_params`, `common_sampler`, `common_chat_*`, `common_speculative_*`); `llama-cpp.h`/`ggml-cpp.h` RAII wrappers. +- CLI: unified `llama` binary (`../../llama.cpp/app/llama.cpp`: `server`, `cli`, hidden `completion/bench/batched-bench/fit-params/quantize/perplexity/download`), `llama-cli` (`../../llama.cpp/tools/cli/`, which embeds a server context), `llama-completion`, `llama-mtmd-cli`, `llama-tts`, `llama-bench`, `llama-diffusion-cli`. +- HTTP: `llama-server` endpoints `/completion`, `/tokenize`, `/detokenize`, `/apply-template`, `/embedding(s)`, `/reranking`, `/infill`, `/props`, `/slots`, `/metrics`, `/lora-adapters`, `/v1/models`, `/v1/completions`, `/v1/chat/completions` (+`/control`, `/input_tokens`), `/v1/responses`, `/v1/embeddings`, `/v1/messages` (Anthropic), `/v1/audio/transcriptions` (audio-input models only), `/models` (router mode load/unload/download/SSE), MCP tool servers, Web UI (`../../llama.cpp/tools/server/README.md#L473-L2040`, `../../llama.cpp/tools/server/server.cpp#L261`). +- Mobile bindings: Android JNI + Kotlin `InferenceEngine`/`AiChat` with Kotlin `Flow` token stream and GGUF metadata reader (`../../llama.cpp/examples/llama.android/lib/src/main/cpp/ai_chat.cpp`, `../../llama.cpp/examples/llama.android/lib/src/main/java/com/arm/aichat/InferenceEngine.kt`, CMake enabling KleidiAI+OpenMP on arm64-v8a `../../llama.cpp/examples/llama.android/lib/src/main/cpp/CMakeLists.txt`); iOS/macOS/visionOS/tvOS `llama.xcframework` via `../../llama.cpp/build-xcframework.sh` and Swift sample `../../llama.cpp/examples/llama.swiftui/llama.cpp.swift/LibLlama.swift` (direct C API from Swift: `llama_batch_init`, `llama_decode`, `llama_sampler_chain_*`). +- Python: only conversion/tooling (`gguf-py`); no runtime Python binding in tree (third-party `llama-cpp-python` is not vendored). [Verified] `../../llama.cpp/README.md` (no bindings list at this commit). +- Distributed: `rpc-server` (`../../llama.cpp/tools/rpc/README.md`). + +## 7. Execution flow + +### (a) Text LLM decode (single sequence, e.g. `examples/simple`) + +```mermaid +sequenceDiagram + participant App as App (simple.cpp) + participant L as libllama (llama-context.cpp) + participant B as llama_batch_allocr + participant M as llama_memory_i (KV cache) + participant G as llm_graph_context / model.build_graph + participant S as ggml_backend_sched + participant BE as Backends (GPU..., CPU) + App->>L: llama_backend_init(); ggml_backend_load_all() + App->>L: llama_model_load_from_file(gguf, mparams) + L->>BE: enumerate devices, place weights (mmap / upload) + App->>L: llama_init_from_model(model, cparams) + L->>M: model.create_memory() + L->>S: ggml_backend_sched_new(); sched_reserve() (pp + tg worst-case graphs) + App->>L: llama_tokenize(prompt) + loop each llama_decode(batch) + App->>L: llama_decode(ctx, batch) + L->>B: balloc->init(batch) validate seq_id/pos/outputs + L->>M: memory_update(false); init_batch(balloc, n_ubatch) + M-->>L: memory context (ubatches + slot_info per ubatch) + loop each ubatch + L->>M: mctx->apply() (apply_ubatch: write cells) + alt graph reusable (can_reuse) + L->>L: reuse gf_res_prev + else + L->>G: build_graph(gparams) + G-->>L: ggml_cgraph + L->>S: sched_reset(); sched_alloc_graph(gf) + S->>S: split_graph (assign backends, splits, copies) + end + L->>L: res->set_inputs(ubatch) (tokens, pos, kq_mask, k/v idxs) + L->>S: ggml_backend_sched_graph_compute_async(gf) + S->>BE: per split: copy inputs, graph_compute_async, record event + L->>BE: ggml_backend_tensor_get_async(logits -> host) + end + App->>L: llama_sampler_sample(smpl, ctx, -1) + L->>L: llama_synchronize(); apply sampler chain on logits + App->>L: llama_token_to_piece(); next llama_batch_get_one(token) + end +``` + +### (b) TTS path (`llama-tts` with Qwen3-TTS or Pocket-TTS) + +```mermaid +sequenceDiagram + participant T as tools/tts/tts.cpp + participant H as mtmd_helper gen_audio pipeline (mtmd-helper-gen.cpp) + participant L as libllama (backbone talker) + participant MT as libmtmd (mtmd.cpp / clip.cpp) + participant SPK as clip graph: speaker encoder + participant GC as clip graph: GEN_CODE (code_predictor / flow net) + participant GW as clip graph: GEN_WAV (code2wav / Mimi decoder) + T->>L: load backbone GGUF (LLM_ARCH_QWEN3TTS / POCKETTTS) + T->>MT: mtmd_init_from_file(mmproj) -> gen_audio_get_info(type, sample_rate) + T->>MT: mtmd_helper_bitmap_init_from_file(speaker.wav/mp3) (miniaudio decode) + T->>H: set_input(prompt, lang, speaker_ref, top_k/top_p/seed) + H->>MT: mtmd_tokenize + encode speaker chunk + MT->>SPK: ECAPA-TDNN (qwen3tts-spkenc) / Mimi encoder (pockettts-spkenc) + SPK-->>H: speaker embedding row(s) + loop step_prompt(n_batch) until 0 + H->>L: llama_decode(prompt tokens + speaker embd via llama_batch.embd) + end + T->>L: sampled = common_sampler_sample(); h_state = llama_get_embeddings_ith(-1) + loop step_gen(sampled, h_state) until EOS / max frames + H->>MT: mtmd_gen_audio_process(GEN_CODE, code0=sampled, embd=h_state) + MT->>GC: code_predictor: 15 more codebooks (on-device top-k/top-p sampling) | flow-matching latent + EOS score + GC-->>H: codes / feats, embd (next backbone input), is_eos + H->>H: buffer codes; every window_frames (72) -> flush_gen_wav + H->>MT: mtmd_gen_audio_process(GEN_WAV, codes, state_data) + MT->>GW: code2wav (Qwen3-TTS) | Mimi/SEANet decoder (Pocket-TTS), state carried across calls + GW-->>H: PCM samples + new state + H->>L: llama_decode(next backbone input embedding) + T->>L: sample next codec_0 token + end + T->>H: get_output() -> WAV bytes (write_wav16) +``` + +## 8. Tests and examples + +- [Verified] Unit tests: `../../llama.cpp/tests/` (71 `llama_build_and_test`/`llama_test` registrations in `../../llama.cpp/tests/CMakeLists.txt`), run with `ctest` after `cmake -B build -DLLAMA_BUILD_TESTS=ON && cmake --build build`; labels `main` (`ctest -L main`), e.g. `test-backend-ops.cpp` (per-op cross-backend correctness/`perf`/`grad`/`support` modes, CSV output used for `docs/ops.md`), `test-tokenizer-0/1-bpe/1-spm`, `test-quantize-fns/perf`, `test-alloc`, `test-batch-alloc`, `test-thread-safety`, `test-save-load-state`, `test-state-restore-fragmented`, `test-recurrent-state-rollback`, `test-mtmd-c-api.c`, `test-mtmd-impl.cpp`, `test-backend-sampler`, `test-rpc-multi-server`, `test-chat*`, `test-jinja`, `test-grammar*`, `test-llama-archs`. Debugging guide: `../../llama.cpp/docs/development/debugging-tests.md` (`scripts/debug-test.sh`). CONTRIBUTING requires running `test-backend-ops` after ggml changes (`../../llama.cpp/CONTRIBUTING.md#L48`). +- [Verified] Server tests: pytest scenarios in `../../llama.cpp/tools/server/tests/` (`tests.sh`, `../../llama.cpp/tools/server/tests/README.md`). +- [Verified] CI: `.github/workflows` includes `build-android.yml`, `build-apple.yml` (builds xcframework, `xcodebuild` of `llama.swiftui`, runs `ctest -L main`), `build-and-test-snapdragon.yml` (Hexagon/OpenCL presets), `build-opencl.yml` (Windows Arm64 Adreno), `build-vulkan.yml`, `build-webgpu.yml`, `build-cann.yml`, `build-openvino.yml`, `build-riscv.yml`, `build-wasm.yml`, `server.yml`, `update-ops-docs.yml`; self-hosted `ggml-ci` via `../../llama.cpp/ci/run.sh` (`../../llama.cpp/ci/README.md`). +- [Verified] Examples relevant to edge/TTS/vision/audio: `../../llama.cpp/tools/tts/tts.cpp` (TTS), `../../llama.cpp/tools/mtmd/mtmd-cli.cpp` (vision/audio chat), `../../llama.cpp/examples/llama.android/` (Kotlin app + JNI lib), `../../llama.cpp/examples/llama.swiftui/` (iOS), `../../llama.cpp/examples/simple/simple.cpp`, `../../llama.cpp/examples/simple-chat/`, `../../llama.cpp/examples/batched/batched.cpp`, `../../llama.cpp/examples/parallel/parallel.cpp`, `../../llama.cpp/examples/embedding/`, `../../llama.cpp/examples/retrieval/`, `../../llama.cpp/examples/diffusion/diffusion-cli.cpp`, `../../llama.cpp/examples/speculative-simple/`, `../../llama.cpp/examples/eval-callback/` (tensor observation via `cb_eval`), `../../llama.cpp/tools/llama-bench/`, `../../llama.cpp/tools/batched-bench/`, `../../llama.cpp/tools/rpc/`. Multimodal model docs: `../../llama.cpp/docs/multimodal.md`, `../../llama.cpp/docs/multimodal/`. +- [Verified] Snapdragon build helper: `../../llama.cpp/scripts/snapdragon/build.py` (Docker toolchain, `--push` to adb). + +## 9. TTS relevance + +**What exists.** [Verified] +- Tool: `llama-tts` (`../../llama.cpp/tools/tts/tts.cpp`, README `../../llama.cpp/tools/tts/README.md`): `llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF -p "Hello world" --output out.wav`, options `--tts-lang` (zh/en/de/it/pt/es/ja/ko/fr/ru), `--tts-speaker-file` (wav/mp3 reference voice), `-n` frames, standard sampling and `-ngl/-b/-ub`. README states the tool "used to serve as a demo for OuteTTS, but it was converted to a more model-agnostic tool". +- Supported pipelines (`enum mtmd_gen_audio_type`, `../../llama.cpp/tools/mtmd/mtmd.h#L370-L374`): `QWEN3TTS` (24 kHz; backbone `LLM_ARCH_QWEN3TTS` reusing the Qwen3-VL graph, `../../llama.cpp/src/models/qwen3tts.cpp`; speaker encoder ECAPA-TDNN `../../llama.cpp/tools/mtmd/models/qwen3tts-spkenc.cpp`; `code_predictor` producing 15 additional codebooks with on-graph top-k/top-p sampling `../../llama.cpp/tools/mtmd/models/qwen3tts-gen.cpp`; `code2wav` vocoder with 72-frame windows and carried state) and `POCKETTTS` (Kyutai Pocket-TTS: CALM backbone without lm_head `../../llama.cpp/src/models/pockettts.cpp`, flow-matching latent generator + EOS head `../../llama.cpp/tools/mtmd/models/pockettts-gen.cpp`, Mimi/SEANet codec encoder+decoder `pockettts-seanet.cpp`, `pockettts-spkenc.cpp`; requires a speaker file). Conversion classes: `../../llama.cpp/conversion/qwen3tts.py` (`Qwen3TTSTalkerModel`, `Qwen3TTSSpeakerEncoderModel`), `../../llama.cpp/conversion/pockettts.py`. +- Legacy path: `LLM_ARCH_WAVTOKENIZER_DEC` (`../../llama.cpp/src/models/wavtokenizer-dec.cpp`: posnet/convnext decoder, group norm, conv1d) and `conversion/wavtokenizer.py` remain in libllama (OuteTTS-era vocoder), but no OuteTTS driver exists in `tools/tts` anymore. `LLM_ARCH_TALKIE` (`../../llama.cpp/src/models/talkie.cpp`) is a 13B text model class ([Unknown] its exact purpose; name suggests a speech LLM but nothing audio-specific is in its graph file header). +- API: experimental, stateless core `mtmd_gen_audio_process(ctx, mtmd_gen_inp{type=GEN_CODE|GEN_WAV, code0, embd, top_k, top_p, seed, temp, codes/feats, state_data}, mtmd_gen_out{codes/feats, embd, is_eos, audio, n_samples, state_data})` (`../../llama.cpp/tools/mtmd/mtmd.h#L385-L438`), and stateful helper `mtmd_helper_gen_audio_{init,set_input,step_prompt,step_gen,get_output}` with a C++ `mtmd_helper::gen_audio` wrapper (`../../llama.cpp/tools/mtmd/mtmd-helper.h#L189-L291`). Design rules for porting new TTS models are documented in `../../llama.cpp/tools/mtmd/README-dev.md#L38-L90` (3-stage pipeline, sidecar models in mmproj, `a.gen.*` tensor prefixes, most logic in `mtmd-helper-gen.cpp`). +- Audio ops available in ggml relevant to vocoders: `CONV_TRANSPOSE_1D`, `IM2COL`, `COL2IM_1D`, `PAD_REFLECT_1D`, `GROUP_NORM`, `CUMSUM`, `TOP_K`, `ARGSORT`, `SSM_CONV`, `TIMESTEP_EMBEDDING` (used by Pocket-TTS flow net time embedding). Backend coverage for these varies (see `../../llama.cpp/docs/ops.md`: e.g. `COL2IM_1D` only SYCL/Vulkan; `CONV_TRANSPOSE_1D` CPU/CANN/CUDA/Metal/SYCL/Vulkan; not OpenCL/HTP), so on Adreno/Hexagon those ops fall back to CPU. +- Audio input for ASR/audio-LLMs: Whisper-style encoders, Qwen2/3-Audio, Ultravox, Voxtral, Granite-Speech, Parakeet, Gemma-4 audio, Qwen2.5/3-Omni (audio+vision input) (`../../llama.cpp/docs/multimodal.md#L108-L138`, projector enum `../../llama.cpp/tools/mtmd/clip-impl.h#L445-L507`). +- Timing instrumentation: `llama-tts` reports prompt / generation / vocoder seconds and frames (`tts.cpp#L188-L201`). + +**What is missing / limitations.** +- [Verified] No streaming audio output in the CLI (single WAV at end) and no server endpoint for speech synthesis (`/v1/audio/speech` absent; only `/v1/audio/transcriptions`). +- [Verified] Only two generation pipelines; OuteTTS driver removed; no Kokoro/CosyVoice/Orpheus/SNAC/Dia/Chatterbox etc. in tree. +- [Verified] `mtmd_gen_audio` API is marked experimental / subject to breaking changes; hparams for code2wav are hard-coded in `clip.cpp` (`conversion/qwen3tts.py#L257`). +- [Verified] Single-sequence TTS driver; no batching of multiple utterances, no cross-request scheduling; state (`c2w_state`) is a byte blob managed by the caller. +- [Inferred] Latency-critical stage 2/3 graphs run through `clip_ctx`'s own `ggml_backend_sched` with per-call graph rebuild (`clip_image_batch_encode` resets and reallocates each call), so no graph reuse/CUDA-graph benefit on the vocoder path. +- [Unknown] No published RTF/latency numbers for TTS in docs. + +## 10. VLA relevance + +**What exists.** [Verified] +- Vision encoders: ~60 projector/encoder types in `clip.cpp` (`../../llama.cpp/tools/mtmd/clip-impl.h#L445-L507`), including SigLIP (`tools/mtmd/models/siglip.cpp`), Qwen2/2.5/3-VL mergers with M-RoPE, Gemma3/3n/4 (MobileNetV5 `mobilenetv5.cpp`), Pixtral, InternVL, Kimi-VL, LLaVA/LLaVA-UHD, MiniCPM-V, Idefics3/SmolVLM, LFM2-VL, PaddleOCR/DeepSeek-OCR, Llama-4, Janus-Pro, CogVLM, Step3-VL, Hunyuan-VL, Nemotron-V2-VL, YouTu-VL, MiMo-VL, Granite-Vision. Encoder-only use is possible via `mtmd_encode`/`clip_image_batch_encode` + `mtmd_get_output_embd` without decoding (embedding extraction), and `clip_init` can be called standalone. +- Numeric / continuous inputs: `llama_batch.embd` lets an application inject arbitrary pre-computed input embeddings (rows of `n_embd_inp`) instead of tokens (`../../llama.cpp/include/llama.h#L262-L270`); `mtmd_bitmap_init_from_audio(n_samples, float*)` accepts raw float sequences (audio); no generic "sensor" tensor input. +- Encoder-decoder and non-causal graphs: `llama_encode`, `llama_set_causal_attn`, T5 (`src/models/t5.cpp`), BERT-family embeddings, classification heads (`llama_model_n_cls_out`, `llama_model_cls_label`), pooling types. +- Diffusion / iterative denoising in text: Dream / LLaDA / LLaDA-MoE / RND1 masked-diffusion LMs driven by `examples/diffusion` (`../../llama.cpp/examples/diffusion/README.md`; algorithms origin/entropy/margin/random/confidence; timestep or block scheduling). Flow matching in mtmd: Pocket-TTS `clip_graph_pockettts_gen` (`../../llama.cpp/tools/mtmd/models/pockettts-gen.cpp`, `time_embed`, `modulate` = adaLN-style conditioning) shows a working pattern for a small flow-matching head on top of a backbone hidden state, generating continuous 32-d latents per step. +- Low-latency loop primitives: graph reuse + CUDA graphs for 1-token steps; backend samplers to avoid host round-trips; `llama_decode` on embeddings; state snapshot/restore (`llama_state_seq_*`); recurrent-state rollback (`n_rs_seq`); abort callback; threadpool pinning/priorities. +- Training hooks: `llama_opt_init/llama_opt_epoch` and `ggml-opt.cpp` (AdamW/SGD) for fine-tuning in ggml (`../../llama.cpp/include/llama.h#L1609-L1625`). + +**What is missing.** [Verified unless noted] +- No VLA model architecture (no OpenVLA, pi0/pi0.5, SmolVLA, GR00T, RT-2 style heads): grep for `proprio|openvla|smolvla|pi0|action head|diffusion policy` finds nothing in `src/`, `tools/`, `conversion/`, `docs/`. +- No action head abstraction: `libllama` outputs are logits (`n_vocab`), embeddings (`n_embd`/pooled), classification (`n_cls_out`), or backend-sampled tokens; a continuous action regression head would need a new arch class in `src/models/` or a sidecar graph in mtmd (as Pocket-TTS does). +- No proprioception/state tokenizer or projector; `llama_batch.embd` requires the caller to compute embeddings externally. +- No diffusion/flow policy sampler loop at library level (Pocket-TTS's is model-specific inside `mtmd-helper-gen.cpp`). +- No structured multi-stage scheduler for a control loop (perception encoder -> backbone -> action decoder) beyond the sequential helper used for TTS; no fixed-rate/real-time scheduling. +- [Inferred] Vision encoder latency on mobile GPUs depends on backend op coverage (`CONV_2D`, `IM2COL`, `UPSCALE`, `PAD`, 2D-RoPE); OpenCL/HTP have gaps in `docs/ops.md`, so parts of encoders may fall back to CPU. +- [Proposal] A VLA port would: (1) convert the VLM backbone via `conversion/` + mmproj; (2) add a `PROJECTOR_TYPE_*` for the vision tower if new; (3) add a sidecar "action expert" graph in `tools/mtmd/models/` driven by a new `mtmd_gen_*`-style stateless process API and a helper driver mirroring `mtmd-helper-gen.cpp`; (4) feed proprio via `llama_batch.embd` rows after a small projector computed in the sidecar graph. + +## 11. Edge deployment profile + +- **Build system.** [Verified] CMake (`../../llama.cpp/CMakeLists.txt`, `../../llama.cpp/ggml/CMakeLists.txt`), presets `CMakePresets.json`, Nix flake, Makefile shim. Static builds via `-DBUILD_SHARED_LIBS=OFF` (`../../llama.cpp/docs/build.md#L63`, `#L335`), `GGML_STATIC`, `GGML_LTO`. Optional components can be disabled: `LLAMA_BUILD_{COMMON,TESTS,TOOLS,EXAMPLES,SERVER,APP,UI}`, `LLAMA_OPENSSL=OFF`, `LLAMA_SUBPROCESS`, `LLAMA_LLGUIDANCE`, `GGML_OPENMP`, `GGML_LLAMAFILE`, `GGML_CPU_REPACK`, `GGML_BACKEND_DL`. `LLAMA_BUILD_MTMD` builds libmtmd standalone (`../../llama.cpp/CMakeLists.txt#L256`). +- **Dependencies.** [Verified] Core (`ggml` + `libllama`) has no external runtime dependencies beyond libc/libstdc++ and optional pthreads/OpenMP. Vendored: cpp-httplib, nlohmann/json, miniaudio, stb_image, sheredom (`../../llama.cpp/vendor/`). Optional: OpenSSL/BoringSSL (HTTPS downloads), CURL (not required; `download.cpp` uses httplib), ffmpeg binaries for video/WebP (runtime, subprocess), backend SDKs (CUDA, ROCm, Vulkan SDK + glslc at build time, OpenCL headers/ICD, Hexagon SDK, CANN, OpenVINO, Dawn for WebGPU, KleidiAI fetched when enabled). +- **Binary size.** [Unknown] No size numbers in docs. [Inferred] CPU-only footprint is dominated by per-arch kernel variants (`GGML_CPU_ALL_VARIANTS` multiplies) and `unicode-data.cpp` (7 k lines of tables); Metal embeds shader source; Vulkan embeds SPIR-V for 180 shaders (times variants); OpenCL embeds 178 kernels; CUDA fat binaries for multiple archs are large (limit `CMAKE_CUDA_ARCHITECTURES`). +- **Platforms.** + - [Verified] Apple Silicon: Metal on by default (`GGML_METAL_DEFAULT ON` when `APPLE`, `../../llama.cpp/ggml/CMakeLists.txt#L95`), Accelerate/BLAS; iOS 16.4+, macOS 13.3+, visionOS 1.0+, tvOS 16.4+ xcframework (`../../llama.cpp/build-xcframework.sh#L7-L10`), `GGML_METAL_EMBED_LIBRARY` for app bundles. No CoreML/ANE backend. + - [Verified] Android: NDK arm64-v8a cross-compile with `GGML_NATIVE=OFF GGML_OPENMP=OFF GGML_LLAMAFILE=OFF LLAMA_OPENSSL=OFF` (`../../llama.cpp/docs/android.md`), KleidiAI for SME2/i8mm/dotprod, OpenCL for Adreno (`../../llama.cpp/docs/build.md#L712-L760`), Hexagon HTP NPU via Snapdragon toolchain containers with `GGML_HEXAGON=ON GGML_OPENCL=ON` preset (`../../llama.cpp/docs/backend/snapdragon/README.md`), Vulkan (generic). No NNAPI, no QNN; Adreno prebuilt kernel library is a manual download. + - [Verified] Embedded NVIDIA (Jetson): CUDA backend; the CMake default arch list does not include sm_87 explicitly (`../../llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt#L28-L54`), so a `native` build on-device or explicit `-DCMAKE_CUDA_ARCHITECTURES=87` is needed ([Inferred]). `GGML_CUDA_GRAPHS` benefits small-batch decode. Unified memory documented (`../../llama.cpp/docs/build.md#L285`). + - [Verified] Generic ARM Linux CPU: NEON/dotprod/i8mm/SVE/SME runtime dispatch, repack, KleidiAI; RISC-V RVV incl. SpacemiT (`../../llama.cpp/docs/build-riscv64-spacemit.md`); s390x; LoongArch. + - [Verified] Windows on Arm (Snapdragon X): OpenCL Adreno and Hexagon docs (`../../llama.cpp/docs/backend/snapdragon/windows.md`). + - [Verified] Browser/WASM: WebGPU via Emscripten (`../../llama.cpp/docs/build.md#L806-L821`), `build-wasm.yml`. +- **Model conversion path and constraints.** [Verified] HF safetensors -> `convert_hf_to_gguf.py` (per-arch class must exist in `conversion/`; unsupported archs fail) -> optional `llama-quantize` (with `llama-imatrix` for IQ/low-bit types) -> optional `gguf-split`. Multimodal encoders need a separate `--mmproj` conversion; TTS sidecars (code2wav, Mimi) must live in the mmproj GGUF (`README-dev.md#L61-L80`). Pre-quantized GGUFs downloadable with `-hf` (HF cache in `common/hf-cache.cpp`). Constraints: no ONNX/TorchScript import; tokenizer must be one of the supported vocab types with a recognized pre-tokenizer (`convert_hf_to_gguf_update.py` maintains hashes); quantized weight repacking happens at load (`use_extra_bufts`), not offline, and is CPU-only. +- **Memory fit tooling.** [Verified] `llama-fit-params` / `common/fit.cpp` and `no_alloc` metadata-only loads (`llama_model_params.no_alloc`) estimate memory before allocating; `llama_get_memory_breakdown` (`../../llama.cpp/src/llama-ext.h#L91`). +- **Runtime knobs for edge.** [Verified] `-ngl`, `--device`, `-t/-tb`, `--cpu-mask/--prio/--poll`, `-c` (KV size), `-ctk/-ctv` (KV quant), `-fa`, `-b/-ub`, `--load-mode` (mmap/mlock/none), `--override-tensor`, `--no-repack`/`use_extra_bufts`, `--kv-unified`. + +## 12. Limitations and unknowns + +1. [Verified] No paged/blocked KV cache; sequences occupy fixed slots in a ring buffer (`n_ctx_seq` each unless unified); long-context multi-tenant serving relies on prompt cache save/restore and context shift. +2. [Verified] No dedicated prefill/decode disaggregation or priority scheduling in `llama-server`; single worker thread, FIFO with deferral. +3. [Verified] Cancellation within a running `llama_decode` only works on CPU (and Metal via abort callback); GPU backends run the whole split. +4. [Verified] Graph is rebuilt per ubatch unless reusable; `clip.cpp` encoder/generator graphs are rebuilt and reallocated on every call. +5. [Verified] TTS API is experimental; only Qwen3-TTS and Pocket-TTS; CLI-only; no streaming HTTP; code2wav hparams hard-coded in `clip.cpp`. +6. [Verified] No VLA/action-head/proprioception support; continuous inputs only via `llama_batch.embd`. +7. [Verified] Op coverage on mobile accelerators is partial (OpenCL, HTP columns in `docs/ops.md`); vocoder/conv ops frequently fall back to CPU on those backends. +8. [Verified] No NNAPI/QNN/CoreML/TensorRT backends; Hexagon requires the proprietary Hexagon SDK and signed DSP libraries; Adreno binary kernel lib is a manual download. +9. [Verified] Python is conversion-only; runtime bindings are out of tree. +10. [Unknown] Binary sizes, per-platform performance, TTS real-time factor, Hexagon vs OpenCL relative speed: no numbers in this tree (benchmarks in `../../llama.cpp/benches/` were not audited). +11. [Unknown] Whether pipeline-parallel copies (`n_copies=4`) apply on single-device mobile builds: [Inferred] no, since `pipeline_parallel` requires more than one non-CPU device (`llama-context.cpp#L428`). +12. [Inferred] The `talkie` and `muse-glimmer` archs are recent additions whose exact application is not documented in-tree. +13. [Unknown] ET backend (`ggml-et`) maturity and availability of hardware. + +## 13. Reference index + +| Path | Role | +|---|---| +| `../../llama.cpp/AGENTS.md` | Contributor/agent policy (read-first) | +| `../../llama.cpp/README.md` | Backend table, docs index | +| `../../llama.cpp/CMakeLists.txt` | Top-level build options (`LLAMA_BUILD_*`, `LLAMA_BUILD_MTMD`) | +| `../../llama.cpp/ggml/CMakeLists.txt` | ggml backend/SIMD options (`GGML_*`) | +| `../../llama.cpp/ggml/include/ggml.h` | tensor types, `ggml_op` enum, custom ops | +| `../../llama.cpp/ggml/include/ggml-backend.h` | backend/device/buffer/sched public API | +| `../../llama.cpp/ggml/include/ggml-cpu.h` | CPU backend API, threadpool, feature queries | +| `../../llama.cpp/ggml/src/ggml.c` | tensor/graph core | +| `../../llama.cpp/ggml/src/ggml-backend.cpp` | `ggml_backend_sched` (split/assign/compute) | +| `../../llama.cpp/ggml/src/ggml-backend-impl.h` | backend vtables, `GGML_BACKEND_DL_IMPL` | +| `../../llama.cpp/ggml/src/ggml-backend-reg.cpp` | registry, dynamic loading | +| `../../llama.cpp/ggml/src/ggml-backend-meta.cpp` | meta device (tensor parallel) | +| `../../llama.cpp/ggml/src/ggml-backend-dl.h` | dlopen helpers | +| `../../llama.cpp/ggml/src/ggml-alloc.c` | graph/tensor allocators | +| `../../llama.cpp/ggml/src/ggml-quants.c` | reference quant/dequant | +| `../../llama.cpp/ggml/src/gguf.cpp` | GGUF reader/writer | +| `../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c` | threadpool, `type_traits_cpu`, graph compute | +| `../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp` | CPU device/reg, extra buffer types, `supports_op` | +| `../../llama.cpp/ggml/src/ggml-cpu/ops.cpp` | CPU op kernels incl. mul_mat quantized path | +| `../../llama.cpp/ggml/src/ggml-cpu/repack.cpp` | runtime weight repacking (Q4_0x4/x8, Q4_Kx8, ...) | +| `../../llama.cpp/ggml/src/ggml-cpu/traits.h` | `extra_buffer_type`/`tensor_traits` interfaces | +| `../../llama.cpp/ggml/src/ggml-cpu/kleidiai/kleidiai.cpp` | Arm KleidiAI microkernel integration | +| `../../llama.cpp/ggml/src/ggml-cpu/amx/amx.cpp` | Intel AMX buffer type | +| `../../llama.cpp/ggml/src/ggml-cpu/llamafile/sgemm.cpp` | tinyBLAS sgemm | +| `../../llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu` | CUDA backend, `supports_op`, CUDA graphs | +| `../../llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt` | CUDA arch selection | +| `../../llama.cpp/ggml/src/ggml-cuda/mmq.cuh` | quantized matmul (int8 MMA) | +| `../../llama.cpp/ggml/src/ggml-cuda/mmvq.cu` | quantized matvec | +| `../../llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp` | Metal device/backend registration | +| `../../llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m` | Metal library compile / metallib load | +| `../../llama.cpp/ggml/src/ggml-metal/ggml-metal-context.m` | Metal command encoding, abort callback | +| `../../llama.cpp/ggml/src/ggml-metal/kernels/` | Metal shader sources | +| `../../llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp` | Vulkan backend | +| `../../llama.cpp/ggml/src/ggml-vulkan/CMakeLists.txt` | shader generation at build | +| `../../llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp` | GLSL->SPIR-V generator | +| `../../llama.cpp/ggml/src/ggml-opencl/ggml-opencl.cpp` | OpenCL/Adreno backend | +| `../../llama.cpp/ggml/src/ggml-opencl/cl-program-cache.cpp` | OpenCL program binary cache | +| `../../llama.cpp/ggml/src/ggml-opencl/kernels/` | OpenCL kernels | +| `../../llama.cpp/ggml/src/ggml-hexagon/ggml-hexagon.cpp` | Hexagon HTP host-side backend | +| `../../llama.cpp/ggml/src/ggml-hexagon/htp/` | DSP-side HVX/HMX kernels | +| `../../llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp` | RPC backend protocol | +| `../../llama.cpp/ggml/src/ggml-opt.cpp` | optimizer/training | +| `../../llama.cpp/include/llama.h` | public C API | +| `../../llama.cpp/src/llama.cpp` | model load entry, device selection | +| `../../llama.cpp/src/llama-ext.h` | extended (non-stable) API | +| `../../llama.cpp/src/llama-model-loader.cpp` | GGUF loading, splits, mmap | +| `../../llama.cpp/src/llama-mmap.cpp` | mmap/mlock | +| `../../llama.cpp/src/llama-model.cpp` | tensors, buffer placement, `create_memory`, `build_graph` | +| `../../llama.cpp/src/models/models.h` | per-arch model classes | +| `../../llama.cpp/src/models/qwen3tts.cpp` | Qwen3-TTS backbone | +| `../../llama.cpp/src/models/pockettts.cpp` | Pocket-TTS backbone | +| `../../llama.cpp/src/models/wavtokenizer-dec.cpp` | WavTokenizer decoder (legacy vocoder) | +| `../../llama.cpp/src/models/talkie.cpp` | Talkie arch | +| `../../llama.cpp/src/models/t5.cpp` | encoder-decoder example | +| `../../llama.cpp/src/llama-arch.h` | arch enum | +| `../../llama.cpp/src/llama-context.cpp` | context, decode/encode, sched, state | +| `../../llama.cpp/src/llama-batch.h` | ubatch splitting | +| `../../llama.cpp/src/llama-batch.cpp` | batch allocator | +| `../../llama.cpp/src/llama-memory.h` | memory interface | +| `../../llama.cpp/src/llama-kv-cache.h` | KV cache class | +| `../../llama.cpp/src/llama-kv-cache.cpp` | KV cache impl (find_slot, apply_ubatch, shift) | +| `../../llama.cpp/src/llama-kv-cells.h` | cell bookkeeping | +| `../../llama.cpp/src/llama-kv-cache-iswa.h` | SWA dual cache | +| `../../llama.cpp/src/llama-memory-recurrent.h` | recurrent state memory | +| `../../llama.cpp/src/llama-graph.h` | graph inputs, params, reuse | +| `../../llama.cpp/src/llama-graph.cpp` | graph building blocks | +| `../../llama.cpp/src/llama-vocab.cpp` | tokenizers | +| `../../llama.cpp/src/unicode-data.cpp` | unicode tables | +| `../../llama.cpp/src/llama-sampler.cpp` | samplers | +| `../../llama.cpp/src/llama-grammar.cpp` | GBNF grammar sampling | +| `../../llama.cpp/src/llama-quant.cpp` | quantization | +| `../../llama.cpp/src/llama-cparams.h` | `LLAMA_MAX_SEQ` | +| `../../llama.cpp/common/speculative.h` | speculative decoding API | +| `../../llama.cpp/common/chat.h` | chat templates / tool calls | +| `../../llama.cpp/common/jinja/README.md` | Jinja engine | +| `../../llama.cpp/common/peg-parser.cpp` | PEG output parser | +| `../../llama.cpp/tools/mtmd/mtmd.h` | mtmd C API incl. audio generation | +| `../../llama.cpp/tools/mtmd/mtmd-helper.h` | helper API (eval chunks, gen_audio) | +| `../../llama.cpp/tools/mtmd/mtmd.cpp` | tokenizer/encoder glue, `mtmd_gen_audio_process` | +| `../../llama.cpp/tools/mtmd/mtmd-helper.cpp` | media decoding, `llama_decode` chunk driver | +| `../../llama.cpp/tools/mtmd/mtmd-helper-gen.cpp` | TTS stateful pipelines | +| `../../llama.cpp/tools/mtmd/mtmd-audio.cpp` | mel/FFT preprocessing | +| `../../llama.cpp/tools/mtmd/mtmd-image.cpp` | image preprocessing | +| `../../llama.cpp/tools/mtmd/clip.cpp` | encoder graphs and execution | +| `../../llama.cpp/tools/mtmd/clip.h` | clip API | +| `../../llama.cpp/tools/mtmd/clip-impl.h` | projector type enum | +| `../../llama.cpp/tools/mtmd/models/` | per-model encoder/generator graphs | +| `../../llama.cpp/tools/mtmd/models/qwen3tts-gen.cpp` | code predictor + on-graph sampling | +| `../../llama.cpp/tools/mtmd/models/qwen3tts-spkenc.cpp` | ECAPA-TDNN speaker encoder | +| `../../llama.cpp/tools/mtmd/models/pockettts-gen.cpp` | flow-matching latent generator | +| `../../llama.cpp/tools/mtmd/models/pockettts-seanet.cpp` | Mimi/SEANet codec | +| `../../llama.cpp/tools/mtmd/models/whisper-enc.cpp` | Whisper encoder | +| `../../llama.cpp/tools/mtmd/models/siglip.cpp` | SigLIP vision encoder | +| `../../llama.cpp/tools/mtmd/models/mobilenetv5.cpp` | Gemma3n vision tower | +| `../../llama.cpp/tools/mtmd/mtmd-cli.cpp` | multimodal CLI | +| `../../llama.cpp/tools/mtmd/README.md` | multimodal usage | +| `../../llama.cpp/tools/mtmd/README-dev.md` | libmtmd design + TTS porting checklist | +| `../../llama.cpp/tools/tts/tts.cpp` | TTS CLI | +| `../../llama.cpp/tools/tts/README.md` | TTS usage | +| `../../llama.cpp/tools/server/server.cpp` | route registration | +| `../../llama.cpp/tools/server/server-context.cpp` | slots, `update_slots`, endpoints impl | +| `../../llama.cpp/tools/server/server-queue.cpp` | task queue | +| `../../llama.cpp/tools/server/server-http.cpp` | HTTP/SSE | +| `../../llama.cpp/tools/server/server-models.cpp` | router mode | +| `../../llama.cpp/tools/server/README.md` | server docs/endpoints | +| `../../llama.cpp/tools/server/tests/README.md` | server pytest | +| `../../llama.cpp/tools/cli/README.md` | CLI args | +| `../../llama.cpp/tools/rpc/rpc-server.cpp` | RPC server | +| `../../llama.cpp/tools/rpc/README.md` | RPC usage | +| `../../llama.cpp/app/llama.cpp` | unified `llama` binary | +| `../../llama.cpp/examples/simple/simple.cpp` | minimal decode loop | +| `../../llama.cpp/examples/batched/batched.cpp` | multi-sequence batching | +| `../../llama.cpp/examples/parallel/parallel.cpp` | serving simulation | +| `../../llama.cpp/examples/diffusion/diffusion-cli.cpp` | diffusion LM driver | +| `../../llama.cpp/examples/diffusion/README.md` | diffusion params | +| `../../llama.cpp/examples/llama.android/lib/src/main/cpp/ai_chat.cpp` | Android JNI | +| `../../llama.cpp/examples/llama.android/lib/src/main/cpp/CMakeLists.txt` | Android lib build | +| `../../llama.cpp/examples/llama.android/lib/src/main/java/com/arm/aichat/InferenceEngine.kt` | Kotlin API | +| `../../llama.cpp/examples/llama.swiftui/llama.cpp.swift/LibLlama.swift` | Swift wrapper | +| `../../llama.cpp/examples/llama.swiftui/README.md` | iOS sample | +| `../../llama.cpp/build-xcframework.sh` | Apple xcframework build | +| `../../llama.cpp/docs/xcframework.md` | xcframework usage | +| `../../llama.cpp/docs/build.md` | build docs (all backends, KleidiAI, OpenCL, Android) | +| `../../llama.cpp/docs/android.md` | Android build | +| `../../llama.cpp/docs/ops.md` | op support matrix | +| `../../llama.cpp/docs/ops/CPU.csv` | CPU op CSV | +| `../../llama.cpp/docs/ops/Vulkan.csv` | Vulkan op CSV | +| `../../llama.cpp/docs/multimodal.md` | multimodal models | +| `../../llama.cpp/docs/multi-gpu.md` | split modes, tensor parallel | +| `../../llama.cpp/docs/speculative.md` | speculative decoding | +| `../../llama.cpp/docs/backend/OPENCL.md` | Adreno OpenCL | +| `../../llama.cpp/docs/backend/snapdragon/README.md` | Hexagon/Snapdragon build | +| `../../llama.cpp/docs/backend/snapdragon/windows.md` | Windows on Snapdragon | +| `../../llama.cpp/docs/backend/snapdragon/developer.md` | Hexagon dev notes | +| `../../llama.cpp/docs/backend/ET.md` | ET backend | +| `../../llama.cpp/docs/backend/SYCL.md` | SYCL | +| `../../llama.cpp/docs/backend/VirtGPU.md` | VirtGPU | +| `../../llama.cpp/docs/build-riscv64-spacemit.md` | RISC-V | +| `../../llama.cpp/docs/development/HOWTO-add-model.md` | adding models | +| `../../llama.cpp/docs/development/debugging-tests.md` | test debugging | +| `../../llama.cpp/docs/autoparser.md` | output parser | +| `../../llama.cpp/tests/CMakeLists.txt` | test registration | +| `../../llama.cpp/tests/test-backend-ops.cpp` | per-op backend tests | +| `../../llama.cpp/ci/README.md` | CI overview | +| `../../llama.cpp/ci/run.sh` | ggml-ci script | +| `../../llama.cpp/convert_hf_to_gguf.py` | conversion entry | +| `../../llama.cpp/conversion/base.py` | conversion base classes | +| `../../llama.cpp/conversion/qwen3tts.py` | Qwen3-TTS conversion | +| `../../llama.cpp/conversion/pockettts.py` | Pocket-TTS conversion | +| `../../llama.cpp/conversion/wavtokenizer.py` | WavTokenizer conversion | +| `../../llama.cpp/gguf-py/gguf/gguf_writer.py` | GGUF writer | +| `../../llama.cpp/gguf-py/gguf/constants.py` | GGUF keys/arch constants | +| `../../llama.cpp/gguf-py/README.md` | gguf-py docs | +| `../../llama.cpp/scripts/create_ops_docs.py` | ops.md generator | +| `../../llama.cpp/scripts/snapdragon/build.py` | Snapdragon build helper | +| `../../llama.cpp/grammars/README.md` | GBNF guide | +| `../../llama.cpp/vendor/` | vendored third-party libs | +| `../../llama.cpp/CONTRIBUTING.md` | contribution/testing rules | diff --git a/docs/engines/mnn.md b/docs/engines/mnn.md new file mode 100644 index 000000000..d9317ca50 --- /dev/null +++ b/docs/engines/mnn.md @@ -0,0 +1,561 @@ +# MNN (alibaba/MNN) — Engine Analysis + +| Field | Value | +|---|---| +| Local path | `/data/zhoutaichang/embedding_infer/MNN` | +| Remote | `https://github.com/alibaba/MNN.git` (origin) | +| Branch / HEAD | `master` @ `bef71b9756a2c77549eddbe33eb97290e3b16602` (2026-09-04, "[Vulkan:Perf] Optimize INT4 cooperative matrix path") | +| Dirty status | clean (`git status --porcelain` empty) | +| Submodules | none (`.gitmodules` absent; third-party code is vendored under `3rd_party/` or fetched by CMake at build time — see §11) | +| Engine version | 3.6.1 ([MNNDefine.h](../../MNN/include/MNN/MNNDefine.h)) | +| Analysis date | 2026-09-08 | +| Reading restrictions | Per [AGENTS.md](../../MNN/AGENTS.md), `schema/private/` and `source/internal/` were **not** read or cited. | + +Evidence labels: **[Verified]** = code/doc read; **[Inferred]** = reasoned from structure; **[Proposal]** = design suggestion; **[Unknown]** = could not determine. Links are relative to `analysis/engines/`. + +--- + +## 0. Summary + +- **What it is** [Verified]: MNN is a C++11 inference engine (no training focus, `-fno-rtti -fno-exceptions`) with two APIs — the low-level Session API (`Interpreter → createSession → runSession`) and the recommended Express/Module API (`Module::load → onForward(VARP)`) — plus a separate transformer stack (`transformers/llm/engine`, `transformers/diffusion/engine`) built on top of Module. See [AGENTS.md](../../MNN/AGENTS.md), [Interpreter.hpp](../../MNN/include/MNN/Interpreter.hpp), [Module.hpp](../../MNN/include/MNN/expr/Module.hpp). +- **Execution runtime vs serving stack** [Verified]: the core is a single-graph, single-request runtime. There is no request scheduler, no continuous batching, no paging. Serving is thin: `mls_server` (`POST /chat/completions`, SSE) and `apps/mnncli` (`/v1/chat/completions`, `/v1/messages`) wrap one `Llm` behind a mutex ([mls_server.cpp](../../MNN/transformers/llm/engine/app/mls_server.cpp), [mnncli_server.cpp](../../MNN/apps/mnncli/src/mnncli_server.cpp)). +- **Model format** [Verified]: FlatBuffers `.mnn` ([MNN.fbs](../../MNN/schema/default/MNN.fbs), 186 `OpType` enumerators) with large weights split into a sidecar `.mnn.weight` addressed by `Op.externalPath` + per-op `external:[offset,size,...]`. Converter frontends: ONNX, TF, TFLite, Caffe, TorchScript, MNN-JSON ([cli.cpp](../../MNN/tools/converter/source/common/cli.cpp)). +- **Backends in tree** [Verified]: 16 dirs under [source/backend](../../MNN/source/backend): `cpu` (+`arm82` fp16, x86 AVX2/AVX512-VNNI, RISC-V RVV, KleidiAI, SME2, bf16), `metal`, `opencl`, `vulkan` (image or buffer variant chosen at build), `opengl`, `cuda`, `musa`, `tensorrt`, `coreml`, `nnapi`, `qnn`, `hiai`, `hexagon`, `rknn` (plugin op), `neuropilot` (stub only in OSS). No LoongArch, no WebGPU. +- **Heterogeneous fallback** [Verified]: every non-CPU `Session` carries a CPU "backup" backend; per-op `Backend::onCreate` returning `nullptr` triggers a retry on CPU inside `_createExecutions` ([Pipeline.cpp#L583-L596](../../MNN/source/core/Pipeline.cpp#L583)), with `WrapExecution` inserting layout/device copies. Only works when the runtime's `CompilerType` is `Compiler_Loop` (full geometry decomposition); NPU-style backends (`CoreML`, `NNAPI`, `QNN`, `HIAI`, `TensorRT`) compile the whole subgraph and are exempt from wrapping. +- **Kernel dispatch** [Verified]: per-backend `addCreator(OpType, Creator*)` maps; CPU registers 75 op types, Metal 36, OpenCL 39 (×buffer/image), Vulkan buffer 35, CUDA ~44. Geometry decomposition lowers shape-manipulation ops into `OpType_Raster` regions so backends only need a small compute subset ([GeometryComputer.cpp](../../MNN/source/geometry/GeometryComputer.cpp), [docs/contribute/backend.md](../../MNN/docs/contribute/backend.md)). +- **LLM quantization** [Verified]: exporter default W4 block-64 asymmetric with fp16 scales; W2/W3/W4/W8/FP16 supported (W2/W3 need ARMv8.6 i8mm); optional HQQ/AWQ/GPTQ/OmniQuant/SmoothQuant ([llmexport.py](../../MNN/transformers/llm/export/llmexport.py)). Runtime dequant is fused into the GEMM under `MNN_LOW_MEMORY` (weight-dequant float GEMM with `memory=normal`, or dynamic-int8-activation GEMM with `memory=low`) ([ConvolutionFloatFactory.cpp](../../MNN/source/backend/cpu/compute/ConvolutionFloatFactory.cpp)). KV cache: int8 / TQ3 / TQ4 on CPU, int8 on Metal, optional mmap-to-disk. +- **LLM engine** [Verified]: `Llm` (text), `Omni : Embedding : Llm` (vision/audio/talker), `Talker : Llm` (speech). Decode is a plain autoregressive loop in `ArGeneration::generate` ([generate.cpp#L44](../../MNN/transformers/llm/engine/src/speculative_decoding/generate.cpp#L44)) with optional lookahead / MTP / EAGLE / DFlash speculative strategies. Modules are cloned per (seq-len-bucket, all_logits) key to avoid resizing (`mPrefillKey=100`, decode key 1). +- **TTS is first-class** [Verified]: (1) in-engine Qwen2.5-Omni thinker→talker→DiT(flow ODE)→BigVGAN and Qwen3-TTS talker→code-predictor→speech-decoder (single waveform callback after all frames, `Talker::generateQwen3TTS` in `omni.cpp`) while Qwen2.5-Omni streams **chunked PCM** through `setWavformCallback(const float*, size_t, bool last)` and a two-thread async DiT/vocoder pipeline ([omni.cpp](../../MNN/transformers/llm/engine/src/omni.cpp)); (2) a native `MNNTTSSDK` (Bert-VITS2, Supertonic, Piper) under [apps/frameworks/mnn_tts](../../MNN/apps/frameworks/mnn_tts); (3) a full sherpa-onnx port (VITS/Kokoro/Matcha/HiFiGAN + Zipformer ASR) under [apps/frameworks/sherpa-mnn](../../MNN/apps/frameworks/sherpa-mnn). +- **VLA is absent** [Verified]: exhaustive grep for VLA/OpenVLA/π0/SmolVLA/LeRobot/diffusion-policy/proprio/action-head yields zero real hits. Building blocks exist: 16 vision-encoder exporters (Qwen2/2.5/3/3.5-VL, Gemma3/4, InternVL, MiniCPM-V, SmolVLM/Idefics3, LFM2-VL, MobileCLIP, ...), M-RoPE, DiT/flow-matching runtime (SD3.5/Sana/Wan2.1 T2V, token2wav DiT). +- **Edge profile** [Verified]: minimal `libMNN.so` links only pthread/dl (Linux) or log/m/android (Android); OpenCL/Vulkan are `dlopen`ed by default; repo-reported core size ≈ 800 KB (Android armv7a, `c++_shared`), 12 MB full iOS static lib; `MNN_BUILD_MINI` strips geometry/shape for fixed-shape models. Platforms with build scripts: Android (NDK, API ≥21 arm64 / 14 armv7), iOS/macOS (CocoaPods `MNN.podspec`, fat framework), Linux/Windows/OpenHarmony, WASM (emcmake), RISC-V (SpacemiT K3), CUDA sm60–120 (incl. sm87 Jetson Orin), Qualcomm QNN/Hexagon, Huawei HIAI, Rockchip RKNN, Apple ANE via CoreML. +- **No JIT / graph capture** [Verified]: kernels are AOT (Metal source strings compiled at first use with a persisted tuning cache; Vulkan SPIR-V arrays; OpenCL online-compiled with binary+tuning cache; CUDA nvcc). No `cudaGraph`, no `MTLIndirectCommandBuffer`. GPU batching is controlled by `OP_ENCODER_NUMBER_FOR_COMMIT`. +- **Bindings** [Verified]: C++ only in core (no C API); Python `MNN.llm` (`response/generate/…`, no TTS callback), Android JNI + Kotlin apps, iOS ObjC++/Swift wrapper with audio-out callback. +- **Concurrency model** [Verified]: one `Llm` = one `Executor` + one `RuntimeManager`; thread-safe only if one thread drives it ([test_multi_instance.cpp](../../MNN/transformers/llm/engine/test/test_multi_instance.cpp)). Cancellation of text decode has no engine API — apps step `generate(1)` themselves; `USER_CANCEL` is set only when the wav callback returns `false`. +- **Top facts for edge TTS/VLA**: TTS with RTF reporting exists today (chunk-streamed for Qwen2.5-Omni, single-callback for Qwen3-TTS, on CPU/GPU, DiT forced fp32 on GPU); the DiT ODE loop is host-driven Express code, so a diffusion action head would be implementable the same way; the missing pieces for VLA are proprio/action I/O and any low-latency control loop. + +--- + +## 1. Architecture overview + +```mermaid +graph TD + subgraph Export["Export / conversion (host)"] + HF["HF checkpoint"] --> LLMEXP["llmexport.py
utils/model_mapper.py, vision.py, audio.py, talker.py, token2wav.py"] + ONNX["ONNX / TF / TFLite / Caffe / TorchScript"] --> CONV["MNNConvert
tools/converter (optimizer/merge, postconvert)"] + LLMEXP --> CONV + GGUF["GGUF / safetensors"] --> G2M["gguf2mnn.py / safetensors2mnn.py"] --> CONV + CONV --> MNNF["llm.mnn + llm.mnn.weight + config.json + tokenizer.txt + embeddings_bf16.bin
(+ visual.mnn, audio.mnn, talker.mnn, dit.mnn, bigvgan.mnn ...)"] + end + subgraph App["Application layer"] + DEMO["llm_demo / qwen3_tts_demo / llm_bench"] + MLS["mls_server / mnncli (HTTP, SSE)"] + PY["pymnn MNN.llm"] + AND["Android MnnLlmChat (JNI) / iOS MNNLLMChat / TaoAvatar"] + end + subgraph LLM["transformers/llm/engine"] + LLMC["Llm (llm.cpp): load, forwardRaw, generate, response, prompt cache"] + OMNI["Omni : Embedding : Llm (omni.cpp): vision/audio encoders, M-RoPE"] + TALK["Talker : Llm: codec LM -> DiT -> BigVGAN / speech_decoder
async token2wav workers, wav callback"] + TOK["Tokenizer (SP / tiktoken / BERT / HF BPE / Pipeline) + jinja.hpp"] + SAMP["Sampler (greedy/topK/topP/minP/tfs/typical/penalty/mixed)"] + GEN["Generation strategies: AR, Lookahead, MTP, EAGLE, DFlash"] + KVM["KVMeta (add/remove/reserve, prefix file)"] + DISK["DiskEmbedding (bf16/int8/int4/int2/int3 rows from file)"] + end + subgraph Express["express/ (Module API)"] + MOD["Module::load / clone -> NetModule -> PipelineModule -> StaticModule"] + EXEC["Executor / RuntimeManager (shared Runtime, hints, cache, external file)"] + EXPR["Expr / VARP lazy graph, ComputeCache"] + end + subgraph Core["source/core (Session API)"] + INTERP["Interpreter -> Schedule::schedule"] + SESS["Session (ModeGroup, resize/run)"] + PIPE["Pipeline: encode (shape+geometry) -> allocMemory (create Execution, wrap copies) -> execute"] + GEO["GeometryComputer (-> Raster/Loop) + SizeComputer"] + BUF["BufferAllocator (Eager / Defer), KVCacheManager"] + BK["Backend / Runtime registry (MNNInsertExtraRuntimeCreator)"] + end + subgraph Backends["source/backend/*"] + CPU["CPU (+arm82 fp16, AVX2/512, RVV, KleidiAI, SME2, bf16)"] + GPU["Metal | OpenCL | Vulkan | OpenGL | CUDA | MUSA | TensorRT"] + NPU["CoreML | NNAPI | QNN | HIAI | Hexagon | RKNN(plugin) | NeuroPilot(stub)"] + end + MNNF --> LLMC + DEMO --> LLMC + MLS --> LLMC + PY --> LLMC + AND --> LLMC + LLMC --> OMNI --> TALK + LLMC --> TOK + LLMC --> SAMP + LLMC --> GEN + LLMC --> KVM + LLMC --> DISK + LLMC --> MOD + MOD --> EXEC + MOD --> EXPR + MOD --> SESS + INTERP --> SESS --> PIPE + PIPE --> GEO + PIPE --> BUF + PIPE --> BK + BK --> CPU + BK --> GPU + BK --> NPU + GPU -. "unsupported op -> CPU backup" .-> CPU +``` + +### Key modules + +| Module | Path | Responsibility | Key symbols | +|---|---|---|---| +| Public headers | [include/MNN](../../MNN/include/MNN) | Session API, tensor, forward types, Express/Module API | `Interpreter`, `Tensor`, `MNNForwardType`, `BackendConfig`, `ScheduleConfig`, `Express::Module`, `Express::Executor`, `VARP` | +| Core scheduling | [source/core/Schedule.cpp](../../MNN/source/core/Schedule.cpp) | Pick backend type, build per-pipeline op lists, mark I/O tensors | `Schedule::schedule`, `Schedule::getAppropriateType`, `Schedule::BackendCache`, `OpCacheInfo` | +| Session | [source/core/Session.cpp](../../MNN/source/core/Session.cpp) | Owns pipelines, translates `SessionMode`/`HintMode` into `RuntimeHint`, resize/run | `Session::Session`, `Session::resize`, `Session::run`, `ModeGroup::setHint`, `createPipelineBackend` | +| Pipeline | [source/core/Pipeline.cpp](../../MNN/source/core/Pipeline.cpp) | shape+geometry → command buffer → execution creation (with CPU fallback) → memory plan → execute | `Pipeline::encode`, `Pipeline::allocMemory`, `_createExecutions`, `_InsertCopy`, `Pipeline::execute`, `_pushTuningTask` | +| Backend abstraction | [source/core/Backend.hpp](../../MNN/source/core/Backend.hpp), [Backend.cpp](../../MNN/source/core/Backend.cpp) | Backend/Runtime interfaces, creator registry, hints | `Backend::onCreate/onAcquire/onCopyBuffer/onSync`, `Runtime`, `RuntimeHint`, `MNNInsertExtraRuntimeCreator`, `registerBackend` | +| Memory | [source/core/BufferAllocator.hpp](../../MNN/source/core/BufferAllocator.hpp) | Static/dynamic pools, mmap-backed allocators | `EagerBufferAllocator`, `DeferBufferAllocator`, `BufferAllocator::createMmap` | +| Copy/wrap | [source/core/WrapExecution.cpp](../../MNN/source/core/WrapExecution.cpp) | Insert device/layout conversion between executions | `WrapExecution::needWrap`, `makeCopyExecution` | +| Geometry | [source/geometry](../../MNN/source/geometry) | Decompose ops into Raster/Loop primitives | `GeometryComputer`, `GeometryComputerUtils::shapeComputeAndGeometryTransform`, `registerGeometryOps` | +| Shape | [source/shape](../../MNN/source/shape) | Output-shape inference per op | `SizeComputer`, `SizeComputerSuite`, `REGISTER_SHAPE` | +| KV cache core | [source/core/KVMeta.hpp](../../MNN/source/core/KVMeta.hpp), [KVCacheManager.hpp](../../MNN/source/core/KVCacheManager.hpp) | Host-side KV cursor; base KV storage w/ disk mmap | `KVMeta::sync`, `KVCacheConfig`, `KVCacheManager::onAlloc/onRealloc` | +| CPU backend | [source/backend/cpu](../../MNN/source/backend/cpu) | Kernels, thread pool, core detection, attention, KV cache | `CPUBackend::addCreator/onCreate`, `CPURuntime`, `ThreadPool`, `MNNGetCPUInfo`, `CPUAttention`, `CPUKVCacheManager`, `CoreFunctions` | +| ARM fp16 | [source/backend/arm82](../../MNN/source/backend/arm82) | fp16 variant of CPU backend (NC8HW8) | `Arm82Backend`, `Arm82Functions::init` | +| GPU backends | [metal](../../MNN/source/backend/metal), [opencl](../../MNN/source/backend/opencl), [vulkan](../../MNN/source/backend/vulkan), [cuda](../../MNN/source/backend/cuda) | Device runtimes, kernels, tuning caches | `MetalBackend`, `OpenCLBackend`, `VulkanBackend`, `CUDABackend` | +| NPU backends | [coreml](../../MNN/source/backend/coreml), [nnapi](../../MNN/source/backend/nnapi), [qnn](../../MNN/source/backend/qnn), [hiai](../../MNN/source/backend/hiai), [hexagon](../../MNN/source/backend/hexagon), [rknn](../../MNN/source/backend/rknn) | Subgraph compilation / delegation | `CoreMLBackend::buildModel`, `NNAPIBackend::buildModel`, `QnnBackend::finalizeGraph`, `NPUBackend::bulidIRModelAndLoad`, `registerRKNNPlugin` | +| Express | [express/](../../MNN/express) | Lazy VARP graph, Module implementations, RuntimeManager | `Executor::RuntimeManager::createRuntimeManager`, `StaticModule`, `PipelineModule`, `Executor::ComputeCache` | +| Converter | [tools/converter](../../MNN/tools/converter) | Frontends + optimizer passes + weight quant/coding | `MNNConverter.cpp`, `cli.cpp`, `PostConverter.cpp`, `merge/FuseAttention.cpp`, `WeightQuantAndCoding.cpp` | +| LLM export | [transformers/llm/export](../../MNN/transformers/llm/export) | HF → MNN (text/vision/audio/talker/token2wav/draft models) | `llmexport.py`, `utils/model_mapper.py`, `utils/vision.py`, `utils/audio.py`, `utils/talker.py`, `utils/token2wav.py`, `gguf2mnn.py` | +| LLM engine | [transformers/llm/engine](../../MNN/transformers/llm/engine) | Runtime for LLM/VLM/omni/TTS/embedding/reranker | `Llm`, `Omni`, `Talker`, `Embedding`, `LlmConfig`, `Tokenizer`, `Sampler`, `Generation*`, `DiskEmbedding` | +| Diffusion engine | [transformers/diffusion/engine](../../MNN/transformers/diffusion/engine) | SD1.5 (PNDM), SD3.5 (flow), Sana (flow, Qwen3 text enc), Wan2.1 T2V | `Diffusion::createDiffusion`, `StableDiffusion`, `DiffusionSD35`, `SanaDiffusion`, `WanDiffusion` | +| CV / audio libs | [tools/cv](../../MNN/tools/cv), [tools/audio](../../MNN/tools/audio) | OpenCV-like ops on VARP; wav I/O, fbank/whisper/conformer/usm mel | `MNN::CV::resize/imread`, `MNN::AUDIO::load/save/whisper_fbank/conformer_fbank/usm_fbank` | +| Apps | [apps/](../../MNN/apps) | Android/iOS chat apps, TTS SDK, sherpa port, mnncli server, Sana app | `MnnLlmChat`, `MnnTaoAvatar`, `mnn_tts`, `sherpa-mnn`, `mnncli` | +| Python | [pymnn/](../../MNN/pymnn) | Bindings: expr/nn/cv/audio/llm/reranker | `pymnn/src/llm.h` (`PyMNNLLM_methods`) | + +--- + +## 2. Device layer + +### Backend abstraction and selection +- [Verified] `MNNForwardType` ([MNNForwardType.h](../../MNN/include/MNN/MNNForwardType.h)): `CPU=0, METAL=1, CUDA=2, OPENCL=3, AUTO=4, NN=5 (NNAPI on Android / CoreML on iOS / QNN online), OPENGL=6, VULKAN=7, USER_0=8 (HIAI), USER_1=9 (TensorRT), HEXAGON=10, USER_3=11, ALL=12, CPU_EXTENSION=13, MNN_MEMORY_AHARDWAREBUFFER=14, MUSA=15`; offline-convert pseudo-types `MNN_CONVERT_QNN=32, MNN_CONVERT_NEUROPILOT=33, MNN_CONVERT_COREML=34`. `MNNGpuMode` bit-flags: `MNN_GPU_TUNING_{NONE,HEAVY,WIDE,NORMAL,FAST}`, `MNN_GPU_MEMORY_BUFFER (1<<6)`, `MNN_GPU_MEMORY_IMAGE (1<<7)`, `MNN_GPU_RECORD_OP (1<<8)`, `MNN_GPU_RECORD_BATCH (1<<9)` — these are OR'ed into `ScheduleConfig::numThread` for GPU backends (the LLM engine does `numThread |= 64 | 512` for OpenCL, [llm.cpp#L232](../../MNN/transformers/llm/engine/src/llm.cpp#L232)). +- [Verified] `BackendConfig` = `MemoryMode {Normal, High, Low}`, `PowerMode {Normal, High, Low}`, `PrecisionMode {Normal, High, Low, Low_BF16}`, and a **union** of `void* sharedContext` / `size_t flags`. `RuntimeStatus` probes: `STATUS_SUPPORT_FP16`, `_DOT_PRODUCT`, `_POWER_LOW`, `_SIMD_GROUP_REDUCE`, `_FUSED_PROJ`. +- [Verified] `class Backend` ([Backend.hpp](../../MNN/source/core/Backend.hpp)): pure virtuals `onCreate(inputs, outputs, op)` (returns `Execution*` or `nullptr` = unsupported), `onResizeBegin/End`, `onExecuteBegin/End`, `onAcquire(tensor, StorageType)`, `onClearBuffer`, `onCopyBuffer(src, dst)`; helpers `onAcquireBuffer/onReleaseBuffer` store a `MemObj` in the tensor descriptor; `StorageType {STATIC, DYNAMIC, DYNAMIC_SEPERATE, DYNAMIC_IN_EXECUTION}`; GPU hooks `onMapTensor/onUnmapTensor/onSync(MapType, toCpu, tensor)`; `getMetaPtr/setMetaPtr` (KV meta). `class Runtime`: `CompilerType {Compiler_Geometry, Compiler_Origin, Compiler_Loop}` (default Loop), `AllocatorType {Defer, Eager}`, `onCreate(config, origin)`, `onReset`, `onGabageCollect(level)`, `onSetCache/onGetCache`, `onSetCachePath`, `onMeasure`, async-work API (`setAsyncWork/waitAsyncWork/mCancelled`), `onConcurrencyBegin/End`. +- [Verified] `RuntimeHint` ([Backend.hpp](../../MNN/source/core/Backend.hpp)) carries all tunables: `memoryAllocatorType`, `winogradMemoryUsed=3`, `cpuDecreaseRate=50`, `dynamicQuantOption`, `attentionOption=8`, `kvcacheSizeLimit=-1`, `kvcacheDirPath`, `prefixcacheDirPath`, `midMemoryPath`, `weightMemoryPath`, `mmapFileSize=1024 MB`, `useCachedMmap`, `encorderNumForCommit=10`, `initThreadNumber`, `useArmSme2Cores`, `enableKleidiAI`, `cpuIds`, `divisionRatio=41`. `Session::ModeGroup::setHint` ([Session.cpp#L77](../../MNN/source/core/Session.cpp#L77)) is the single translation point from `Interpreter::HintMode`. +- [Verified] Registry: `registerBackend()` in [Backend.cpp](../../MNN/source/core/Backend.cpp) (under `std::call_once`) always registers CPU and conditionally CoreML (`MNN_COREML_ENABLED`), NNAPI, QNN, RKNN plugin, Hexagon, OpenCL, Metal, NeuroPilot. Vulkan, OpenGL, CUDA, MUSA, TensorRT, HIAI self-register via file-scope static initializers (e.g. [VulkanRuntime.cpp](../../MNN/source/backend/vulkan/runtime/VulkanRuntime.cpp), [cuda/Register.cpp](../../MNN/source/backend/cuda/Register.cpp), [GLBackend.cpp](../../MNN/source/backend/opengl/GLBackend.cpp)). `MNNGetExtraRuntimeCreator` instantiates a probe `Runtime` when the creator was registered with `needCheck=true` (OpenCL, Vulkan, HIAI) so a missing driver degrades gracefully. +- [Verified] Discovery/selection: `Schedule::getAppropriateType` ([Schedule.cpp#L111](../../MNN/source/core/Schedule.cpp#L111)) resolves `MNN_FORWARD_AUTO` through the fixed priority list `HIAI → CoreML/NN → TensorRT → CUDA → OpenCL → Metal → Vulkan → CPU`, falls back to `config.backupType` if the creator is missing, and demotes OpenCL+`Power_Low` if `STATUS_SUPPORT_POWER_LOW` is false. `RuntimeFactory::create` is a one-liner over the registry ([RuntimeFactory.cpp](../../MNN/source/core/RuntimeFactory.cpp)). + +### Memory allocation and pools +- [Verified] [BufferAllocator.hpp](../../MNN/source/core/BufferAllocator.hpp): `EagerBufferAllocator` (free-list `multimap`, 64-byte alignment via `MNN_MEMORY_ALIGN_DEFAULT` in [MNNMemoryUtils.h](../../MNN/source/core/MNNMemoryUtils.h), per-thread groups) and `DeferBufferAllocator` (symbolic plan then `compute()/apply()` into one arena). Factories `createDefault`, `createMmap(dir, prefix, postfix, autoRemove, syncValid)`, `createRecurse(parent)`. +- [Verified] CPU: `CPURuntime` owns one static Eager pool and `MNN_CPU_MAX_BUFFER_INDEX=2` dynamic buffers; `createDynamicBufferAlloctor` returns Defer vs Eager based on `hint().memoryAllocatorType` (`HintMode::MEM_ALLOCATOR_TYPE`) ([CPUBackend.cpp](../../MNN/source/backend/cpu/CPUBackend.cpp)). `hint().weightMemoryPath` swaps the static pool for an mmap-backed one (this is `use_mmap` in the LLM config → `EXTERNAL_WEIGHT_DIR`); `midMemoryPath` does the same for dynamic buffers (`EXTERNAL_FEATUREMAP_DIR`). +- [Verified] `SessionMode` memory knobs: `Session_Memory_Collect` runs `onGabageCollect(0)` after every `allocMemory` ([Session.cpp#L363](../../MNN/source/core/Session.cpp#L363)); `Session_Memory_Cache` keeps static memory. `Session_Input_User` (what `Module::Config::shapeMutable=true` maps to) keeps inputs user-owned and stages copies through `inputTensorCopyCache` in `Pipeline::_copyInputs`. +- [Verified] Express default: `Executor::getGlobalExecutor()` creates a 1-thread CPU runtime with `memoryAllocatorType=0` (Defer) ([Executor.cpp#L154](../../MNN/express/Executor.cpp#L154)). + +### Host↔device transfers and synchronization +- [Verified] [Tensor.hpp](../../MNN/include/MNN/Tensor.hpp): `copyFromHostTensor`, `copyToHostTensor`, `map/unmap(MapType, DimensionType)`, `wait(MapType, finish)`. `Tensor::map` calls `Backend::onSync` then `onMapTensor`, falling back to a temp host copy when mapping is unsupported ([Tensor.cpp](../../MNN/source/core/Tensor.cpp)). +- [Verified] Metal ([MetalBackend.mm](../../MNN/source/backend/metal/MetalBackend.mm)): one `MTLCommandQueue`, a rolling `MTLCommandBuffer`, `mEncoderCount` reset at `onExecuteBegin`, encoder flushed/committed every `hint().encorderNumForCommit` ops (`OP_ENCODER_NUMBER_FOR_COMMIT`), `onSync(toCpu)` = flush + commit + `waitUntilCompleted`; `onAcquire` handles `DYNAMIC_IN_EXECUTION` from an execution pool; tuned threadgroup tables persisted through `onGetCache/onSetCache` (`MetalTuneLevel` default `Wide`, schema [MetalCache.fbs](../../MNN/source/backend/metal/schema/MetalCache.fbs)). +- [Verified] OpenCL ([OpenCLBackend.cpp](../../MNN/source/backend/opencl/core/OpenCLBackend.cpp)): `setGpuMode` decodes BUFFER/IMAGE (AUTO → BUFFER on Mali/Intel, IMAGE elsewhere) and tuning level; record-queue (`MNN_GPU_RECORD_OP/BATCH`) when the device supports it; `onSync(toCpu)` = `commandQueue().finish()`; SVM fine-grain buffers under `MNN_OPENCL_SVM_ENABLE`; binary+tuning cache in a FlatBuffer (`CLCache` schema under [opencl/core/runtime](../../MNN/source/backend/opencl/core/runtime)). The buffer path can be compiled out with `MNN_OPENCL_SIZE_CUT`. +- [Verified] Vulkan buffer variant ([VulkanBackend.cpp](../../MNN/source/backend/vulkan/buffer/backend/VulkanBackend.cpp)): `mDirect = !(gpuMode & MNN_GPU_RECORD_BATCH)`; all command buffers submitted in one `vkQueueSubmit` + single `VulkanFence` in `_finish()`; `onCopyBuffer` stages through a host-visible buffer; dynamic pool is an `EagerBufferAllocator` aligned to `nonCoherentAtomSize`. Image variant selected by `MNN_VULKAN_IMAGE=ON` (default) at build time. + +### Portability — what is actually in tree + +| Backend dir | Forward type | Registration | CMake gate | Delegation granularity / notes | +|---|---|---|---|---| +| [cpu](../../MNN/source/backend/cpu) | CPU / CPU_EXTENSION | `registerCPURuntimeCreator` | always | per-op; `Compiler_Loop` | +| [arm82](../../MNN/source/backend/arm82) | CPU_EXTENSION (fp16) | `registerArm82RuntimeCreator` → `Arm82Functions::init` | `MNN_ARM82` (ON) | chosen when `Precision_Low` and `fp16arith` | +| [metal](../../MNN/source/backend/metal) | METAL | `registerMetalRuntimeCreator` | `MNN_METAL` + APPLE | per-op | +| [opencl](../../MNN/source/backend/opencl) | OPENCL | `registerOpenCLRuntimeCreator` (needCheck) | `MNN_OPENCL` | per-op; dlopen unless `MNN_USE_SYSTEM_LIB` | +| [vulkan](../../MNN/source/backend/vulkan) | VULKAN | static init (needCheck) | `MNN_VULKAN`, `MNN_VULKAN_IMAGE` | per-op; dlopen `libvulkan` | +| [opengl](../../MNN/source/backend/opengl) | OPENGL | static init | `MNN_OPENGL` | per-op; links GLESv3/EGL | +| [cuda](../../MNN/source/backend/cuda) | CUDA | static init | `MNN_CUDA` (+`MNN_CUDA_QUANT/BF16/TUNE_PARAM`) | per-op; sm 60–120 | +| [musa](../../MNN/source/backend/musa) | MUSA=15 | static init | `MNN_MUSA` (+ stub/compat/native) | Moore Threads | +| [tensorrt](../../MNN/source/backend/tensorrt) | USER_1 | static init | `MNN_TENSORRT` | whole subgraph at `onResizeEnd` | +| [coreml](../../MNN/source/backend/coreml) | NN | `registerCoreMLRuntimeCreator` | `MNN_COREML` | whole subgraph: `buildModel()` at `onResizeEnd`, `invokeModel()` at `onExecuteEnd` | +| [nnapi](../../MNN/source/backend/nnapi) | NN | `registerNNAPIRuntimeCreator` | `MNN_NNAPI` (API ≥ 29) | whole graph via `ANeuralNetworksCompilation`; `Compiler_Geometry` | +| [qnn](../../MNN/source/backend/qnn) | NN (online) / CONVERT_QNN (offline) | `registerQNNRuntimeCreator` | `MNN_QNN`, `MNN_QNN_ONLINE_FINALIZE`, `MNN_QNN_CONVERT_MODE` | `Compiler_Origin`; graph finalized once; context binary cached via `onGetCache/onSetCache`; SDK via [prepare_qnn_deps.sh](../../MNN/prepare_qnn_deps.sh) | +| [hiai](../../MNN/source/backend/hiai) | USER_0 | static init (needCheck) | `MNN_NPU` (not an `option()`) | whole graph `bulidIRModelAndLoad`; `Compiler_Origin` | +| [hexagon](../../MNN/source/backend/hexagon) | HEXAGON=10 | `registerHexagon` (probes DSP lib) | `MNN_HEXAGON` | per-op over decomposed stream (`Compiler_Loop`), forced 1 thread | +| [rknn](../../MNN/source/backend/rknn) | (CPU + Plugin op) | `registerRKNNPlugin` | `MNN_RKNN` + `MNN_WITH_PLUGIN` | one plugin op wraps a whole `.rknn` file | +| [neuropilot](../../MNN/source/backend/neuropilot) | CONVERT_NEUROPILOT=33 | — | forced OFF ([CMakeLists.txt](../../MNN/source/backend/neuropilot/CMakeLists.txt)) | not shipped in OSS | + +- [Verified] NNAPI and CoreML both claim `MNN_FORWARD_NN`; `MNNInsertExtraRuntimeCreator` rejects duplicates, so only one can be linked per build [Inferred from registry assert]. +- [Verified] CPU ISA paths: `source/backend/cpu/arm/arm64` (NEON, `*_ARMV82_*` sdot, `*_ARMV86_*` i8mm/smmla, `low_memory/` W2/W3/W4 GEMM, `sme2_asm/`), `arm32`, `x86_x64/{sse,avx,avxfma,avx512}` (AVX512-VNNI split), `riscv/rvv` (+ SpacemiT IME2), `kleidiai/` (fetched v1.16.0 by [cmake/KleidiAI.cmake](../../MNN/cmake/KleidiAI.cmake)), `bf16/`. Runtime feature detection `MNNGetCPUInfo()` ([CPURuntime.cpp](../../MNN/source/backend/cpu/CPURuntime.cpp)): `fp16arith, dot, i8mm, sve2, sme2, rvv/zvfh, groups (big/little clusters by cpufreq), smeCoreNumber, perfCoreNumber`. + +### Multi-backend fallback (unsupported op on GPU) +- [Verified] `Session::createPipelineBackend` ([Session.cpp#L26](../../MNN/source/core/Session.cpp#L26)) creates `cache.first` (target) and, unless the target is CPU, a separate CPU `cache.second` with `flags=4` (`MNN_CPU_USE_DEFAULT_BACKEND`). +- [Verified] `_createExecutions` ([Pipeline.cpp#L538](../../MNN/source/core/Pipeline.cpp#L538)): try `OpCommonUtils::createExecutionWithExternal(mBackend, …)`; if `nullptr`, `// Try Backup` on `mBackupBackend`; if still null → `NOT_SUPPORT`. `_InsertCopy` then walks every command's inputs/outputs and, when `WrapExecution::needWrap(t, curBackend)` is true, appends an `OpType_Copy` command built by `WrapExecution::makeCopyExecution` (const tensors cached via `copyConstCache`). `needWrap` returns false for `MNN_FORWARD_NN` and types ≥ `MNN_CONVERT_QNN`, and for CPU↔CPU_EXTENSION only when `bytes` or `pack` (on NC4HW4 data) differ ([WrapExecution.cpp](../../MNN/source/core/WrapExecution.cpp)). +- [Verified] Coarser fallback under `Session_Backend_Auto`: `Pipeline::allocMemory` asks `Runtime::onMeasure` per op; if any reports `initCostLong` it prints "Turn back to cpu", starts `_pushTuningTask` asynchronously and swaps the whole pipeline backend to CPU ([Pipeline.cpp#L1079](../../MNN/source/core/Pipeline.cpp#L1079)). +- [Verified] Only `Compiler_Loop` runtimes get op-level CPU fallback ([docs/contribute/backend.md](../../MNN/docs/contribute/backend.md)); NPU backends compile the whole graph and refuse (return null) unsupported ops. + +### Runtime sharing +- [Verified] `Executor::RuntimeManager::createRuntimeManager(config)` ([Executor.cpp#L334](../../MNN/express/Executor.cpp#L334)) calls `_getOrCreateRuntime(type, …)` on the current `Executor` — an existing runtime of the same forward type is **reused** (`onReset` with new thread count), and `mRuntime.second` (CPU backup) is always the executor's CPU runtime. `StaticModule::clone` reuses the RuntimeManager's runtime and shared const tensors, so N cloned Modules share one thread pool/allocator/tuning cache while owning separate `Backend`/`Session` objects ([StaticModule.cpp](../../MNN/express/module/StaticModule.cpp)). `Executor::newExecutor` + `ExecutorScope` gives a fully independent runtime (used by the Talker's async worker threads). + +### CPU threading and core binding +- [Verified] `MNN_USE_THREAD_POOL=ON` (default) → own pool ([ThreadPool.cpp](../../MNN/source/backend/cpu/ThreadPool.cpp)): pools keyed by affinity mask, `MNN_THREAD_POOL_MAX_TASKS=2` concurrent task slots, sleeping-worker bitmask. `CPURuntime::_resetThreadPool` may return fewer threads than requested. `CPURuntime::_validateCpuIds` forbids mixing little and big cores; `Power_Low` → little cluster, `Power_High` → highest clusters, `Power_Normal` → no pinning; `MNNSetSchedAffinity` via raw `sched_setaffinity` syscall. Work split by `computeDivideSizes` weights little cores with `CPU_LITTLECORE_DECREASE_RATE` (default 50%). + +--- + +## 3. Kernel layer + +### Operator set and registration +- [Verified] [MNN.fbs](../../MNN/schema/default/MNN.fbs): `enum OpType` with 186 entries; transformer block `Attention=299, FmhaV2=300, Fmhca=301, SeqLen2Spatial=302, SplitGeLU=303, GroupNorm=304, LinearAttention=305, RoPE=306, FusedLinear=307, GatedRMSNorm=308`; quant ops `ConvInt8=513, Int8ToFloat=514, DepthwiseConvInt8=515, FloatToInt8=517, EltwiseInt8=518, DynamicQuant=155`; control `While=600, If=601, LayerNorm=603, GridSample=604`; escape hatches `Plugin=256`, `Extra=512`; `Stft=156` is the only audio-ish op. `table Op { inputIndexes, main:OpParameter, name, outputIndexes, type, defaultDimentionFormat=NHWC, externalPath }`; `table Net { bizCode, extraTensorDescribe, extraInfo, oplists, outputName, preferForwardType, sourceType, tensorName, tensorNumber, usage (INFERENCE/TRAIN/INFERENCE_STATIC), subgraphs, mnn_uuid }`. `AttentionParam { kv_cache, kv_shared_layer, layer_index, kv_shared_layer_index, mhq_quant, output_c4, attnScale }`, `RoPEParam`, `FusedLinearParam`, `LinearAttentionParam` (gate folding), `LoopParam/RegionCommand` (the Loop IR). Other schema files: [CaffeOp.fbs](../../MNN/schema/default/CaffeOp.fbs) (`IDSTQuan`, `Convolution2D`), [TensorflowOp.fbs](../../MNN/schema/default/TensorflowOp.fbs), [Tensor.fbs](../../MNN/schema/default/Tensor.fbs) (`MNN_DATA_FORMAT {NCHW, NHWC, NC4HW4, NHWC4}`, `Blob.external`), [UserDefine.fbs](../../MNN/schema/default/UserDefine.fbs), [TFQuantizeOp.fbs](../../MNN/schema/default/TFQuantizeOp.fbs), [Type.fbs](../../MNN/schema/default/Type.fbs), [ExtraInfo.fbs](../../MNN/schema/default/ExtraInfo.fbs), [TrainInfo.fbs](../../MNN/schema/default/TrainInfo.fbs). +- [Verified] Registration per backend: CPU `REGISTER_CPU_OP_CREATOR(name, OpType)` → `CPUBackend::addCreator` into a `std::map`; generated [CPUOPRegister.cpp](../../MNN/source/backend/cpu/CPUOPRegister.cpp) registers 75 op types (incl. `Attention`, `LinearAttention`, `RoPE`, `Stft`, `Plugin`, `ConvInt8`, `DynamicQuant`, `While`). Metal [MetalOPRegister.mm](../../MNN/source/backend/metal/MetalOPRegister.mm) 36 (transformer ops under `MNN_SUPPORT_TRANSFORMER_FUSE`). OpenCL keys by `pair` (buffer vs image) — [OpenCLOPRegister.cpp](../../MNN/source/backend/opencl/core/OpenCLOPRegister.cpp) 66 entries / 39 types. Vulkan buffer 35 / image 26 (self-registering statics). CUDA ~44 (static `CUDACreatorRegister`). NNAPI 23, CoreML 22, QNN 41. + +### Dispatch mechanism +- [Verified] `Execution` ([Execution.hpp](../../MNN/source/core/Execution.hpp)): `onResize`, `onExecute` (pure), `onClone` (capability probe when `dst==nullptr`), plus a string-keyed `insertExtraCreator/searchExtraCreator`. `CPUBackend::onCreate` ([CPUBackend.cpp](../../MNN/source/backend/cpu/CPUBackend.cpp)) rewrites `Convolution` → `ConvInt8` when I/O `quantAttr.type == DT_INT8` and the conv has quantized weights, then looks up `gCreator`. +- [Verified] Precision → CPU variant (`CPURuntime::onCreate`): `Precision_Low` + `fp16arith` → `Arm82Backend` (pack 8, bytes 2); `Precision_Low_BF16` → BF16 core functions; `flags==4` → plain CPU; else `AVX2Backend` on x86. All variants share the `CoreFunctions` table ([CommonOptFunction.h](../../MNN/source/backend/cpu/compute/CommonOptFunction.h): `pack`, `bytes`, `supportFp16arith/SDot/I8mm/SME2`, kernel pointers) — so **NC4HW4 is really NC{pack}HW{pack}**: 4 (fp32 NEON), 8 (fp16 / AVX2), 16 (AVX512). +- [Verified] The flow per op at resize: `SizeComputer::computeOutputSize` → `GeometryComputer::onCompute` (or identity `DefaultGeometryComputer`) → `Backend::onCreate` → `Execution::onResize`; at run: `Execution::onExecute` in `Pipeline::execute` ([Pipeline.cpp#L1167](../../MNN/source/core/Pipeline.cpp#L1167)). + +### Geometry decomposition +- [Verified] [GeometryComputer.hpp](../../MNN/source/geometry/GeometryComputer.hpp): `GeometryComputer::Context` holds one pre-built `OpType_Raster` op used for every emitted region command; `registerGeometryComputer(computer, types, CompilerType)`; two tables (`Compiler_Geometry`, `Compiler_Loop`), `Compiler_Origin` bypasses geometry. 51 files (`GeometryConv2D`, `GeometryBatchMatMul`, `GeometryReshape`, `GeometryPermute`, `GeometryBinary`, `GeometryReduce`, `GeometryLayernorm`, `GeometryFusedProj`, `GeometryGatedRMSNorm`, …), 84 registrations in [GeometryOPRegister.cpp](../../MNN/source/geometry/GeometryOPRegister.cpp). `GEOMETRY_COMPUTE_MASK` bits: `FUSEREGION`, `FUSEREGION_MULTI`, `USELOOP`, `OPENCACHE`. +- [Verified] Contract ([docs/contribute/op.md](../../MNN/docs/contribute/op.md), [docs/contribute/backend.md](../../MNN/docs/contribute/backend.md)): schema → shape → (optional) geometry → per-backend execution; if geometry is written, no per-backend kernel is required. [Inferred] Minimal backend residue: `Raster`, `Convolution(+Depthwise)`, `MatMul`, `BinaryOp`, `UnaryOp`, `Pooling`, `Reduction`, `Softmax`, `ReLU/PReLU`, `Scale`, `While/Loop` — the intersection of all backend registries. +- [Verified] `Usage_INFERENCE_STATIC` models (converted with `--saveStaticModel`) skip shape+geometry entirely (`needComputeGeometry=false` in `Schedule::schedule`; `Pipeline::encode` copies commands verbatim) — this is what `MNN_SKIPBUILD_GEOMETRY`/`MNN_BUILD_MINI` rely on. + +### Shape inference / dynamic shapes +- [Verified] [SizeComputer.hpp](../../MNN/source/shape/SizeComputer.hpp): `onComputeSize`, `onComputeFlops`, `REGISTER_SHAPE(_INPUTS)` (ops whose shape depends on input *content*, e.g. `Stft`, mark `mNeedContentInputIndex`); 82 files, 122 registrations ([ShapeRegister.cpp](../../MNN/source/shape/ShapeRegister.cpp)); transformer shapes gated by `MNN_SUPPORT_TRANSFORMER_FUSE`. +- [Verified] Runtime shape change: `Interpreter::resizeTensor` → `Session::setNeedResize`; `Session::resize()` ([Session.cpp#L306](../../MNN/source/core/Session.cpp#L306)) re-runs `Pipeline::encode` (shape+geometry) then `allocMemory` (execution creation cached in `executionCache`, memory re-planned). `Session_Resize_Check/Fix` (`Module::traceOrOptimize`) records a resize trace and freezes it. For Express Modules, `shapeMutable=true` → `Session_Input_User`. + +### Layout +- [Verified] `Tensor::DimensionType {TENSORFLOW (NHWC), CAFFE (NCHW), CAFFE_C4 (NC4HW4)}`; `NativeInsideDescribe::dimensionFormat` defaults to `MNN_DATA_FORMAT_NC4HW4` ([TensorUtils.hpp](../../MNN/source/core/TensorUtils.hpp)); `Region` is the raster primitive. `Arm82Backend` reinterprets NC4HW4 as NC8HW8 ([Arm82Backend.hpp](../../MNN/source/backend/arm82/Arm82Backend.hpp)). Layout conversion between ops/backends is automatic via `WrapExecution` (§2). + +### Quantization formats and where dequant happens +- [Verified] Classic PTQ int8: [tools/quantization](../../MNN/tools/quantization) (`quantized.out`, [calibration.cpp](../../MNN/tools/quantization/calibration.cpp): keys `quant_bits`, `feature_quantize_method` (KL/ADMM/EMA), `weight_quantize_method`, `feature_clamp_value`, `weight_clamp_value`, `winogradOpt`, `skip_quant_op_names`). Weights stored as `IDSTQuan { buffer, alpha, type (1 int8 / 2 sparse / 3 fp16 / 4 weightInt8 / 8 external low-bit), quantScale, scaleIn/Out, aMaxOrBits, aMin, readType, alphaFp16, scaleStorage }` ([CaffeOp.fbs](../../MNN/schema/default/CaffeOp.fbs)) and decoded by `ConvolutionCommon::load` ([ConvolutionCommon.cpp](../../MNN/source/core/ConvolutionCommon.cpp)) into `Int8Common { weight, alpha, canUseInt4/Int3/Int2, asymmetric }`. +- [Verified] LLM weight quant (export): `--quant_bit {2,3,4,8,16}` (default 4), `--quant_block` (default 64; 0 = per-channel), `--lm_quant_bit/--lm_quant_block`, `--visual_quant_bit/--visual_quant_block`, `--embed_bit {16,8,4}`, `--sym`, `--scale_bit {16,32}`, `--awq`, `--hqq`, `--gptq_path`, `--omni`, `--smooth`, `--calib_data`, `--act_bit {8,16}`, `--quant_config` (per-op JSON) — [llmexport.py](../../MNN/transformers/llm/export/llmexport.py); packing in [utils/mnn_converter.py](../../MNN/transformers/llm/export/utils/mnn_converter.py) (`quant`, `write_header`, `build_weight`) and [utils/mnn_utils.py](../../MNN/transformers/llm/export/utils/mnn_utils.py) (`repack_low_bits`). MNNConvert equivalents: `--weightQuantBits`, `--weightQuantBlock`, `--weightQuantAsymmetric`, `--hqq`, `--weightQuantScaleBit` ([cli.cpp](../../MNN/tools/converter/source/common/cli.cpp)). +- [Verified] Runtime dequant on CPU (`MNN_LOW_MEMORY`, forced ON by `MNN_BUILD_LLM`): `ConvolutionFloatFactory::_createUnit` ([ConvolutionFloatFactory.cpp](../../MNN/source/backend/cpu/compute/ConvolutionFloatFactory.cpp)) — with `Memory_Low` → `DenseConvInt8TiledExecutor(isDynamicQuant=true)` (activations quantized to int8 per forward: `BatchAsyDynamicQuant`/`BatchSymDynamicQuant`, `DYNAMIC_QUANT_OPTIONS` 1 = per-tensor, 2 = block-quant input), else `DenseConvolutionTiledExecutor` where block-wise scale/bias pointers are threaded into the packed GEMM (`selectLowMemoryMatmulFunc`) — no materialized float weights. ARM64 kernels: [arm64/low_memory](../../MNN/source/backend/cpu/arm/arm64/low_memory) (`MNNGemmInt8AddBiasScale_ARMV82_w{2,3,4}_Unit.S`, `_ARMV86_w{2,3,4}_`, `MNNDynamicQuantFP32_Pack{4,8}.S`). GPU: Metal `MetalConvolution1x1.mm` accepts `mDequantBits ∈ {2,3,4,8}` (Q4 16-byte GEMV); OpenCL `ConvBufLowMemoryExecution`/`ConvLowMemoryExecution` + `gemm_int.cl`/`gemv_conv1x1_buf.cl`; Vulkan buffer `gemv_dequant_int4.comp`, `dynamic_w8a8_coop_gemm_fused.comp`, `VulkanConv1x1CoopA8`; CUDA `weight_only_quant/ConvFpAIntBExecution.cu` + CUTLASS int8. +- [Verified] fp16: `MNN_ARM82` (ON; `Precision_Low`), bf16 `MNN_SUPPORT_BF16` (OFF). KleidiAI: `MNN_KLEIDIAI` (ON) but used only if `RuntimeHint::enableKleidiAI` (`CPU_ENABLE_KLEIDIAI`, default from `MNN_KLEIDIAI_DEFAULT_ON`=OFF) and the conv is 1×1 stride-1. SME2: `MNN_SME2` (ON), hints `CPU_SME2_INSTRUCTIONS`, `CPU_SME2_NEON_DIVISION_RATIO` (default 41). +- [Verified] Attention/KV quant: `RuntimeHint::attentionOption` (default 8): `%8` → 0 none / 1 K int8 / 2 K+V int8 / 3 K TQ3 / 4 K+V TQ3 / 5 K TQ4 / 6 K+V TQ4; `/8` → flash attention on/off ([CPUAttention.cpp](../../MNN/source/backend/cpu/CPUAttention.cpp), `KVQuantMode` in [CPUKVCacheManager.hpp](../../MNN/source/backend/cpu/CPUKVCacheManager.hpp)). LLM config key `attention_mode` (legacy `quant_qkv`). Metal supports only int8 KV and flash attention for `head_dim ∈ {64,128,256}` ([docs/transformers/llm.md](../../MNN/docs/transformers/llm.md)); other GPU backends ignore `attention_mode` [Verified, repo doc]. +- [Verified] Fused transformer ops: `MNN_SUPPORT_TRANSFORMER_FUSE`; converter merge passes in [optimizer/merge](../../MNN/tools/converter/source/optimizer/merge) (`FuseAttention`, `FuseFmhaV2`, `FuseFmhca`, `FuseGeLu`, `FuseSplitGeLu`, `FuseGroupNorm`, `FuseLayerNorm{,V2,V3,RMS,RMSGamm,WithoutGammBeta}`, `MergeDynamicQuantV1/V2`, `ConvertMatMulToConv2D`) and [postconvert/FuseTransformerC4.cpp](../../MNN/tools/converter/source/optimizer/postconvert/FuseTransformerC4.cpp); `RoPE`, `FusedLinear` (QKV / gate-up / LN+proj), `LinearAttention` gate folding, `GatedRMSNorm` are emitted by the exporter (`--disable_fuse_qkv_proj`, `--disable_fuse_gate_up_proj`, `--disable_fuse_ln_proj`, `--disable_transformer_c4`). Cross-layer KV sharing: `AttentionParam.kv_shared_layer_index` → `Execution::onClone` from a registry in `_createExecutions`. + +### Compilation / JIT / graph capture +- [Verified] No machine-code JIT. `codegen/` ([codegen/CMakeLists.txt](../../MNN/codegen/CMakeLists.txt): `MNN_CODEGEN_OPENCL/METAL/CUDA`, all OFF) is source-level op fusion emitted at `Session::resize` when `Session_Codegen_Enable`. Metal shaders are embedded C strings ([AllShader.cpp](../../MNN/source/backend/metal/AllShader.cpp)) compiled by the OS with pipeline-state + tuning cache; Vulkan SPIR-V pre-generated ([buffer/compiler](../../MNN/source/backend/vulkan/buffer/compiler)); OpenCL online-built with `CLCache`; CUDA AOT `.cu`. No `cudaGraph`/`MTLIndirectCommandBuffer` anywhere. `Interpreter::setCacheFile` / `RuntimeManager::setCache` persist tuning across runs (LLM engine writes `tmp_path/mnn_cachefile.bin` for non-CPU backends). + +### Fallback when op unsupported +- [Verified] Per-op CPU backup (§2); if CPU also lacks the op → `NOT_SUPPORT` at resize. Unresolved `OpType_Extra` is a converter-time failure (only Metal registers an `Extra` creator; CPU has none). + +### Extension points: custom op / backend +- [Verified] Plugin ops: `MNN_WITH_PLUGIN`; [include/MNN/plugin](../../MNN/include/MNN/plugin) (`PluginKernel.hpp`: `CPUComputeKernel::init/compute/resize`, `REGISTER_PLUGIN_COMPUTE_KERNEL(name, kernel)`; `PluginShapeInference.hpp`: `REGISTER_PLUGIN_OP(name, inferShapeKernel)`), string-keyed; `OpType_Plugin { type:string, attr }` executed by [CPUPlugin.cpp](../../MNN/source/backend/cpu/CPUPlugin.cpp). RKNN and offline-QNN reuse this path. No `registerCustomOp` API exists. +- [Verified] Native op: schema + `REGISTER_SHAPE` + optional geometry + `REGISTER_{CPU,METAL,OPENCL}_OP_CREATOR` / `VulkanBackend::addCreator` ([docs/contribute/op.md](../../MNN/docs/contribute/op.md)); converter `--customOpLibs`/`--allowCustomOp`. +- [Verified] New backend: subclass `Backend` + `Runtime`, register via `MNNInsertExtraRuntimeCreator(type, creator, needCheck)` ([docs/contribute/backend.md](../../MNN/docs/contribute/backend.md)). + +--- + +## 4. Model runner + +### Model loading / export / conversion +- [Verified] `.mnn` = FlatBuffer `Net`; `Interpreter::createFromFile/createFromBuffer`, `createSession(ScheduleConfig{type, numThread|gpuMode, backendConfig, backupType, saveTensors, path})`, `setSessionMode`, `setSessionHint`, `setCacheFile`, `setExternalFile`, `resizeTensor/resizeSession`, `runSession(WithCallBack)`, `updateSessionToModel` ([Interpreter.hpp](../../MNN/include/MNN/Interpreter.hpp)). `SessionMode` values 0–17 (`Session_Debug/Release`, `Input_Inside/User`, `Output_Inside/User`, `Resize_Direct/Defer`, `Backend_Fix/Auto`, `Memory_Collect/Cache`, `Codegen_Disable/Enable`, `Resize_Check/Fix`, `Module_Forward_Separate/Combine`); `HintMode` 0–17 (see §2); `ExternalPathType {KVCACHE_DIR, FEATUREMAP_DIR, WEIGHT_DIR, NPU_FILE_DIR, PREFIXCACHE_DIR}`. +- [Verified] Module API: `Module::load(inputs, outputs, file|buffer, rtMgr, Config{dynamic=false, shapeMutable=true, rearrange=false, backend, base})` ([Module.hpp](../../MNN/include/MNN/expr/Module.hpp)); implementation `NetModule → PipelineModule (sub-graphs, If/While/MoE modules) → StaticModule (wraps a Session)` ([Module.cpp](../../MNN/express/module/Module.cpp), [PipelineModule.hpp](../../MNN/express/module/PipelineModule.hpp), [StaticModule.cpp](../../MNN/express/module/StaticModule.cpp)). `Module::load` auto-sets the external file to `.weight` if none set. `Module::clone(module, shareParams)` shares weights, new Session. Lazy Express eval: `Executor::LazyMode {LAZY_FULL, LAZY_CONTENT, LAZY_COMPUTE_ONCE}`; `Executor::ComputeCache` in [express/Utils.hpp](../../MNN/express/Utils.hpp). +- [Verified] External weights: `Op.externalPath` + `Convolution2D.external=[offset, weight_bytes, bias_bytes]`, `Blob.external=[offset, bytes]`; writer [writeFb.cpp](../../MNN/tools/converter/source/common/writeFb.cpp) (`--saveExternalData`, auto for large models, threshold `externalTreshold=64 KB` in [config.hpp](../../MNN/tools/converter/include/config.hpp)); loaded lazily by `FileLoader` at execution creation (`createExecutionWithExternal`). +- [Verified] Converter: [MNNConverter.cpp](../../MNN/tools/converter/source/MNNConverter.cpp) → `Cli::convertModel`; frontends `onnx/`, `tensorflow/`, `tflite/`, `caffe/`, `torch/` (TorchScript, `MNN_BUILD_TORCH`); flags `--fp16`, `--weightQuantBits`, `--weightQuantBlock`, `--hqq`, `--optimizeLevel 0/1/2`, `--optimizePrefer`, `--keepInputFormat`, `--saveStaticModel`, `--saveExternalData`, `--transformerFuse`, `--transformerFuseC4`, `--transformerFuseQkvProj/GateUpProj/LnProj`, `--alignDenormalizedValue`, `--detectSparseSpeedUp`, `--customOpLibs`, `--allowCustomOp`, `--testdir/--thredhold`, `--JsonFile`, `--rknn`. Pass pipeline in [PostConverter.cpp](../../MNN/tools/converter/source/optimizer/PostConverter.cpp) (`RemoveUnusefulOp → … → MergeBNToConvolution → … → AddTensorFormatConverter → ReIndexTensor`), 41 merge passes, 33 `onnxextra/` rewrites. +- [Verified] GGUF import ([gguf2mnn.py](../../MNN/transformers/llm/export/gguf2mnn.py)): F16, Q4_0, Q4_1, Q4_K, Q5_0, Q5_1, Q6_K, Q8_0 → MNN low-bit blobs (others assert); [safetensors2mnn.py](../../MNN/transformers/llm/export/safetensors2mnn.py) for pre-quantized/plain safetensors. +- [Verified] LLM export pipeline ([llmexport.py](../../MNN/transformers/llm/export/llmexport.py): `export_talker → export_vision → export_audio → export_eagle → export_dflash → export_language → export_mtp → export_tokenizer → export_config`) produces `llm.mnn`, `llm.mnn.weight`, `llm_config.json`, `config.json` (runtime defaults: `backend_type=cpu, thread_num=4, precision=low, memory=low, sampler_type=mixed, temperature=0.8, top_k=40, top_p=0.9, min_p=0.05, …`), `tokenizer.txt` (or `.mtok` binary), `embeddings_bf16.bin` / `embeddings_int{8,4}.bin`, optional `visual.mnn`, `audio.mnn`, `talker.mnn`, `predit.mnn`, `dit.mnn`, `bigvgan.mnn`, `spk_dict.mnn`, `code_predictor.mnn`, `speech_decoder.mnn`, `speaker_encoder.mnn`, `mtp.mnn`, EAGLE/DFlash draft files, `ple_embeddings_bf16.bin` (Gemma4). Supported families in [model_mapper.py](../../MNN/transformers/llm/export/utils/model_mapper.py): llama, qwen/qwen2/qwen3/qwen3_moe/qwen3_5(_moe/_text), qwen2_vl/qwen2_5_vl/qwen3_vl(_moe), qwen2_audio, qwen3_asr, qwen2_5_omni, qwen3_tts, internlm, mobilellm, baichuan, deepseek-vl, llama4_text, mimo, poi_qwen2_mtp, chatglm/chatglm2, phi/phi-msft, internvl_chat, gemma2/gemma3(_text)/gemma4, openelm, idefics3/smolvlm, llava_qwen2 (FastVLM), hunyuan_v1_dense/hunyuan_vl, gpt_oss, minicpm/minicpmv, funaudiochat, glm_ocr, lfm2/lfm2_moe/lfm2_vl/lfm2_audio. NPU export helpers: [export/npu](../../MNN/transformers/llm/export/npu) (`generate_llm_qnn.py`, `generate_llm_mlda.py`) driving [compilefornpu.cpp](../../MNN/tools/cpp/compilefornpu.cpp). + +### Execution lifecycle (init → warmup → step) +- [Verified] `Llm::createLLM(config.json)` → `Omni` if `is_visual || is_audio || has_talker` else `Llm` ([llm.cpp#L88](../../MNN/transformers/llm/engine/src/llm.cpp#L88)). `Llm::load()` ([llm.cpp#L319](../../MNN/transformers/llm/engine/src/llm.cpp#L319)): check files → clear module pool before re-creating runtime (Backend lifetime) → `initRuntime()` (ScheduleConfig from `backend_type/thread_num/precision/power/memory`, `RuntimeManager`, hints from `setRuntimeHint`) → `Tokenizer::createTokenizer` → merge `context.json` into jinja context → `DiskEmbedding` (+ PLE) → chat template → `Sampler` → `Module::load({input_ids, attention_mask, position_ids, logits_index[, deepstack_embeds, ple_embeddings]}, {logits[, talker_embeds][, hidden_states]})` with `shapeMutable = !(opencl|vulkan|npu)`, `rearrange=true`, external file = `llm_weight` → `setSpeculativeConfig` (spec only if the model declares a dimensioned `logits_index` input) → `GenerationStrategyFactory::create` → clone decode module (key `<1,false>`) and spec module (``) → preallocate mask/position VARPs → `mGenerationStrategy->load` → status `RUNNING`. +- [Verified] **No implicit warmup**. Optional `Llm::tuning(OP_ENCODER_NUMBER, candidates)` benchmarks commit granularity on Metal/CPU (skipped on OpenCL) then resets KV; demos call it with `{1,5,10,20,30,50,100}`. +- [Verified] Step: `Llm::forwardRaw(hidden, mask, pos, extra)` ([llm.cpp#L569](../../MNN/transformers/llm/engine/src/llm.cpp#L569)) picks module by `(seqLenKey = inDecode ? seq_len : mPrefillKey(100), allLogits)`, chooses `logitsAllIdx`/`logitsLastIdx`, `onForward`, validates outputs, optional `Tensor::wait` when `async=false`, then `mMeta->sync()`. `forwardVec` sets `mMeta->add = seq_len` and splits into `size_limit` blocks for chunked prefill. +- [Verified] Runtime hints actually set by the LLM engine ([llm.cpp#L181](../../MNN/transformers/llm/engine/src/llm.cpp#L181)): `INIT_THREAD_NUMBER=4`, `MEM_ALLOCATOR_TYPE=0`, `ATTENTION_OPTION`, `USE_CACHED_MMAP`, `DYNAMIC_QUANT_OPTIONS`, `KVCACHE_INFO` (ptr), `CPU_SME2_NEON_DIVISION_RATIO`, `MMAP_FILE_SIZE`, `OP_ENCODER_NUMBER_FOR_COMMIT` (0 during prefill / 512 during decode on OpenCL); external paths `KVCACHE_DIR` (`kvcache_mmap`), `PREFIXCACHE_DIR`, `WEIGHT_DIR` (`use_mmap`), `NPU_FILE_DIR`. Never set: `KVCACHE_SIZE_LIMIT`, `CPU_CORE_IDS`, `CPU_ENABLE_KLEIDIAI`. + +### Dynamic shapes +- [Verified] Strategy is **bucketed module clones, not per-step resize**: prefill of any length maps to key 100 (the Session inside resizes on shape change since `Session_Input_User`), decode always key 1 (no resize), spec-verify key `draft+1`. Mask/position VARPs are reused when dims match (`needNewVar`). Chunked prefill (`size_limit: [block, 1]`) keeps shapes to a handful of values, needed on OpenCL/Vulkan/NPU where `shapeMutable=false`. + +### State & KV cache +- [Verified] `KVMeta` ([KVMeta.hpp](../../MNN/source/core/KVMeta.hpp)): `block=4096, previous, remove, reserve[] (pairs begin/len), n_reserve, add, file_name, file_flag {NoChange, PendingWrite, PendingRead}, seqlen_in_disk, layer_index, layer_nums, attn_scale, spec_block`; `sync()` applies `previous = previous - remove + add + Σreserve_len` after each forward. `Llm::setKVCacheInfo`, `eraseHistory(begin,end)` (partial rollback via reserve segments), `reset()` (`remove = previous`), `reuse_kv` (multi-turn KV retention), `prompt_cache` (text-prefix diff → only the token delta is prefilled, [llm.cpp#L1167](../../MNN/transformers/llm/engine/src/llm.cpp#L1167)), `setPrefixCacheFile` (KV prefix to disk; **requires `kvcache_mmap=true`**). +- [Verified] Storage: [KVCacheManager.hpp](../../MNN/source/core/KVCacheManager.hpp) (`KVCacheConfig { mKVCacheDir, mPrefixCacheDir, mExpandChunk=64, mBlockNum=1, mKvAlignNum }`, key layout `{numhead, [maxlen/hP, headdim, hP]}` (or int8 `[maxlen/hP8, headdim/lP8, hP8, lP8]`), value `{numhead, [headdim/hP, maxlen, hP]}`, mmap FDs) and [CPUKVCacheManager.cpp](../../MNN/source/backend/cpu/CPUKVCacheManager.cpp) (`onAlloc/onRealloc` grow by `mExpandChunk` on demand, `expandKVCacheInMem/InDisk`, `moveKVCacheFromMemToDisk` when `kvcacheSizeLimit` exceeded, quant via `MNNQuantAttentionKey/Value`). Layout is contiguous per head — **no paging, no eviction policy**; the KV is a growable array trimmed only by explicit `remove/reserve`. Per-`Llm` instance (each cloned Module has its own attention executions → own KV). Metal/OpenCL/Vulkan/CUDA have their own attention executions ([MetalAttention.mm](../../MNN/source/backend/metal/MetalAttention.mm), [AttentionBufExecution.cpp](../../MNN/source/backend/opencl/execution/buffer/AttentionBufExecution.cpp), [VulkanAttention.cpp](../../MNN/source/backend/vulkan/buffer/execution/VulkanAttention.cpp), [AttentionExecution.cu](../../MNN/source/backend/cuda/execution/AttentionExecution.cu)). +- [Verified] Embedding lookup is host-side from a file (`DiskEmbedding`, [diskembedding.cpp](../../MNN/transformers/llm/engine/src/diskembedding.cpp): bf16/int8/int4/int2/int3 rows, buffered `FileLoader` reads, tie-embeddings read from inside `llm.mnn.weight`), then fed as `input_embeds`. + +### Sampling and speculative decoding +- [Verified] [sampler.cpp](../../MNN/transformers/llm/engine/src/sampler.cpp): `greedy, temperature, topK, topP, minP, tfs, typical, penalty (repetition/presence/frequency + n-gram), mixed` pipeline (`penalty` forced first), `logit_bias`, `banned_tokens`, GPU top-k prefilter. +- [Verified] [speculative_decoding](../../MNN/transformers/llm/engine/src/speculative_decoding): `ArGeneration` (plain AR), `LookaheadGeneration` (n-gram lookahead, `draft_predict_length`, `ngram_match_maxlen`, `lookup_file`), `MtpGeneration` (`mtp.mnn`, needs `hidden_states`), `EagleGeneration` (tree drafting: `eagle_model/eagle_fc/eagle_d2t`, `eagle_depth=3`, `eagle_topk`), `DFlashGeneration` (block diffusion-style drafts: `dflash_model/fc/kvmat`, `dflash_block_size=8`). Verification = greedy longest prefix (`Generation::draftVerify`), KV rollback via `KVMeta.reserve/spec_block`. + +### Multimodal stages +- [Verified] `Omni` ([omni.cpp](../../MNN/transformers/llm/engine/src/omni.cpp)): separate `mProcessorRuntimeManager` from the `mllm` sub-config (can put encoders on a different backend/thread count); `visual.mnn` (`mVisionModule`) with per-family preprocess `defaultVisionProcess` (CV resize+normalize), `qwen2VisionProcess` (smart-resize to pixel budget, patch 14 / 16 for Qwen3-VL, window mask), `gemma4VisionProcess`, `smolvlmVisionProcess`, `minicpmVisionProcess`, `hunyuanVisionProcess`, `qwenVideoProcess` (fps sampling, temporal patch 2, `video_max_frames=768`); `audio.mnn` (`mAudioModule`) fed by `whisper_fbank` / `conformer_fbank` / `usm_fbank` (16 kHz; Qwen2-Audio clipped to 3000 frames), audio window mask for 2-input encoders; M-RoPE position streams (`MropeInfo`); prompt syntax `…`, ``, `` or `MultimodalPrompt{images, audios, videos}`; embeddings spliced into pad-token runs in `Omni::embedding`. +- [Verified] `Talker` (§9) for speech output; `Embedding`/reranker (`sentence_embeddings` output pooled inside the graph, [embedding.cpp](../../MNN/transformers/llm/engine/src/embedding.cpp), [reranker.hpp](../../MNN/transformers/llm/engine/include/llm/reranker.hpp)). +- [Verified] Diffusion ([diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/diffusion.hpp)): `DiffusionModelType {STABLE_DIFFUSION_1_5, STABLE_DIFFUSION_TAIYI_CHINESE, SANA_DIFFUSION, WAN2_1_T2V}`; `StableDiffusion` = text_encoder → unet (PNDM/PLMS, [scheduler.hpp](../../MNN/transformers/diffusion/engine/src/scheduler.hpp)) → vae_decoder with module release by `memoryMode` and `Session_Resize_Check/Fix` to freeze the denoise-loop resize plan; `DiffusionSD35` (3 text encoders, DiT transformer, **flow matching** `step_flow_match`); `SanaDiffusion` (Qwen3-0.6B text encoder, DiT, flow matching, img2img; [sana_diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/sana_diffusion.hpp)); `WanDiffusion` (T2V, `runVideo`, frame dump; [wan_diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/wan_diffusion.hpp), [wan_diffusion_demo.cpp](../../MNN/transformers/diffusion/engine/wan_diffusion_demo.cpp)). Export: [diffusion/export](../../MNN/transformers/diffusion/export). Build: `MNN_BUILD_DIFFUSION` (forces `MNN_LOW_MEMORY`, `MNN_SUPPORT_TRANSFORMER_FUSE`, `MNN_BUILD_OPENCV`). + +--- + +## 5. Scheduler + +### (a) Request / token / pipeline-level scheduling +- **Request scheduling: N/A** [Verified]. One `Llm` object serves one conversation at a time; there is no queue, no batching across requests, no continuous batching, no prefill/decode disaggregation. `mls_server` and `mnncli` serialize HTTP requests with a `std::mutex` around `llm->response` ([mls_server.cpp](../../MNN/transformers/llm/engine/app/mls_server.cpp)). Batch dim is fixed to 1 in the exported graph (`input_ids` `[1, seq]`; [llmexport.py](../../MNN/transformers/llm/export/llmexport.py) patches the batch dim). +- **Concurrency** [Verified]: N independent `Llm` instances in N threads work (each has its own `Executor`/`RuntimeManager`; [test_multi_instance.cpp](../../MNN/transformers/llm/engine/test/test_multi_instance.cpp)); `ExecutorScope` makes the executor thread-local. Runtimes created from the *global* executor share the CPU runtime and thread pool (`_getOrCreateRuntime`). `LlmContext::mutex` only protects token vectors for external readers. +- **Prefill vs decode** [Verified]: prefill = one `forwardVec` over all prompt embeddings (optionally chunked by `size_limit` block size — `Llm::generate(ids)` loops `generate(chunk_embeds, 0)`), decode = `ArGeneration::generate` loop of `sample → is_stop → tokenizer_decode → stream to os → forwardVec({token})` ([generate.cpp#L44](../../MNN/transformers/llm/engine/src/speculative_decoding/generate.cpp#L44)). Stage-specific tuning: `Llm::switchMode(Prefill|Decode)` toggles OpenCL commit batching. +- **Multi-stage pipelines** [Verified]: (1) `Omni::responseInterleaved` interleaves one thinker decode step with one talker step in a single loop (`while (!thinker_done || !talker_done)`, [omni.cpp#L2286](../../MNN/transformers/llm/engine/src/omni.cpp#L2286)); (2) `Talker` async token2wav: main thread runs the codec LM and enqueues `WavChunk`s (60 codec tokens + 24/12 context) → `ditWorkerLoop` thread (own `Executor::newExecutor`, cloned `mPreDit/mDit`) → `mMelQueue` → `vocoderWorkerLoop` thread (cloned BigVGAN) → wav callback ([omni.cpp#L2944](../../MNN/transformers/llm/engine/src/omni.cpp#L2944)). Enabled only when the talker's backend is CPU and a processor runtime distinct from the talker runtime exists; GPU backends run token2wav synchronously. `Talker::finalize` waits up to 60 s for the last chunk. +- **Cancellation / abort** [Verified]: `LlmStatus::USER_CANCEL` exists and stops every generation loop, but the engine sets it in exactly one place — when the wav callback returns `false` ([omni.cpp#L3403](../../MNN/transformers/llm/engine/src/omni.cpp#L3403)). Text decode has no cancel API; `response()` blocks until stop/max tokens/`timeout_ms`. Apps work around this by calling `response(..., max_new_tokens=0)` (prefill only) and stepping `llm->generate(1)` per token under their own flag ([llm_session.cpp#L300](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp#L300)). `timeout_ms` config aborts with `TIMEOUT` after prefill or during decode. Async resize tuning can be cancelled through `Runtime::mCancelled`. +- **Backpressure** [Verified]: none for text (stream is an `std::ostream*`, writes are synchronous). For audio, the wav callback's return value is the only backpressure/abort signal; the async queues are unbounded `std::queue`s. + +### (b) Graph / operator / thread-level scheduling +- [Verified] `Schedule::schedule` ([Schedule.cpp#L291](../../MNN/source/core/Schedule.cpp#L291)): builds `ScheduleInfo` = const-tensor init (with external weight loader) + one `PipelineInfo` per `ScheduleConfig` (multi-path sessions supported via `createMultiPathSession`), ops kept in model order (`generateScheduleGraph` filters by `path`/`saveTensors`), marks INPUT/OUTPUT usage, computes `needInputContentForShape`, and for static models disables shape/geometry. +- [Verified] `Pipeline::encode` → `shapeComputeAndGeometryTransform` (shape + geometry + const folding, with `OpResizeCache` for shape-change replay) → `allocMemory` (`_createExecutions` → `_InsertCopy` → `_SetTensorBackend` → per-execution `onResize` between `onResizeBegin/End`, memory planned by the Defer/Eager allocator with `_recycleDynamicMemory`) → `execute` (linear walk of `executeBuffer.command` calling `onExecute`, between `onExecuteBegin/End` on both backends). Op order is topological as emitted by the converter; there is no reordering or multi-stream assignment. +- [Verified] Thread pools: CPU `ThreadPool` per affinity mask, `numThread` per `ScheduleConfig`, big/little weighting, SME2 core selection (`useArmSme2Cores`, `divisionRatio`). GPU "streams": one Metal command queue with commit every N encoders; one OpenCL command queue with optional record queue; one Vulkan submit per `onExecuteEnd`; no multi-stream overlap. Async: `Runtime::setAsyncWork` for background tuning; `INIT_THREAD_NUMBER` for parallel module load. + +--- + +## 6. I/O layer + +- **Image preprocessing** [Verified]: `CV::ImageProcess` ([ImageProcess.hpp](../../MNN/include/MNN/ImageProcess.hpp): 15 formats incl. NV21/NV12/I420, `Filter {NEAREST, BILINEAR, BICUBIC}`, affine `Matrix`, mean/normal, `convert(src → Tensor|raw)`, `draw`), kernels in [ImageProcessFunction.cpp](../../MNN/source/backend/cpu/compute/ImageProcessFunction.cpp). OpenCV-like `MNN::CV` on VARP ([tools/cv/include/cv/cv.hpp](../../MNN/tools/cv/include/cv/cv.hpp): `resize/warpAffine/cvtColor/GaussianBlur/threshold/findContours/imread/imwrite/solvePnP…`), `MNN_BUILD_OPENCV`, `MNN_IMGCODECS` (stb-based). +- **Audio preprocessing** [Verified]: `MNN::AUDIO` ([audio.hpp](../../MNN/tools/audio/include/audio/audio.hpp)): `load/save` (wav), `hamming_window/hann_window`, `melscale_fbanks`, `spectrogram`, `mel_spectrogram`, `fbank`, `whisper_fbank`, `conformer_fbank` (NeMo-style), `usm_fbank` (Gemma4), resamplers (linear, soxr-HQ); `MNN_BUILD_AUDIO`. Python exposes all but the two fork additions ([pymnn/src/audio.h](../../MNN/pymnn/src/audio.h)). +- **Tokenization** [Verified]: in-house [tokenizer.cpp](../../MNN/transformers/llm/engine/src/tokenizer/tokenizer.cpp): `tokenizer.txt` with magic 430 and type `SENTENCEPIECE | TIKTOIKEN | BERT | HUGGINGFACE | PIPELINE` (`.mtok` binary: `Normalizer/PreTokenizer/TokenizerModel/TokenDecoder`, added tokens, byte-level BPE); special/stop tokens in the file; `is_stop()`; chat templates rendered by a vendored single-header Jinja2 engine [jinja.hpp](../../MNN/transformers/llm/engine/src/tokenizer/jinja.hpp) (jinja.cpp, not minja) with tools support; template/eos/bos read from the tokenizer file or `config.json["jinja"]`; `stripThinkBlocks` for prompt-cache stability ([prompt_cache_utils.hpp](../../MNN/transformers/llm/engine/src/prompt_cache_utils.hpp)). +- **Streaming** [Verified]: text tokens are written to `std::ostream*` per token (apps subclass `std::streambuf`, e.g. [llm_stream_buffer.hpp](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_stream_buffer.hpp), with UTF-8 reassembly); no token callback API. Audio: `setWavformCallback(bool(const float* pcm, size_t n, bool last))`, chunked at ~120 mel frames / 60 codec tokens (Qwen2.5-Omni) or once at the end (Qwen3-TTS, single `speech_decoder` call). Perf counters in `LlmContext` (`prefill_us, decode_us, sample_us, ttfa_us, audio_us`). +- **Buffering** [Verified]: disk-streamed embeddings; mmap weights (`use_mmap` → `EXTERNAL_WEIGHT_DIR`, `mmap_size`, `use_cached_mmap`); KV to disk (`kvcache_mmap`); OpenCL/Metal tuning cache file. +- **Postprocessing** [Verified]: detokenize per token; wav writing is the caller's job (`MNN::AUDIO::save(path, VARP, 24000)` in [qwen3_tts_demo.cpp](../../MNN/transformers/llm/engine/demo/qwen3_tts_demo.cpp)); diffusion writes PNG via `MNN::CV::imwrite`. +- **Application boundaries** [Verified]: C++ (`Interpreter`, `Express`, `Llm`; no C API beyond [MNNSharedContext.h](../../MNN/include/MNN/MNNSharedContext.h)); Python `MNN.llm` ([pymnn/src/llm.h](../../MNN/pymnn/src/llm.h): `create/load/forward/generate/response/get_context/set_config/erase_history/txt_embedding/create_lora…`, GIL released in `response`; no TTS); HTTP `mls` (`POST /chat/completions`, SSE) and `mnncli` (`/v1/models`, `/v1/chat/completions`, Anthropic `/v1/messages`); CLIs `llm_demo`, `llm_bench`, `embedding_demo`, `reranker_demo`, `qwen3_tts_demo`, `multi_lora_demo`, `rollback_demo`, `tokenizer_demo`, `apply_template`, `llm_logits_diff` ([demo/](../../MNN/transformers/llm/engine/demo)); Android JNI + Kotlin (`MnnLlmChat`, `MnnTaoAvatar`), iOS ObjC++ wrapper ([LLMInferenceEngineWrapper.h](../../MNN/apps/iOS/MNNLLMChat/MNNLLMiOS/InferenceEngine/LLMInferenceEngineWrapper.h) with `setAudioWaveformCallback`), engine iOS demo ([engine/ios](../../MNN/transformers/llm/engine/ios)); OpenHarmony build only (no app); no WASM/JS binding (build recipe only). + +--- + +## 7. Execution flow + +### (a) Text LLM: one `response()` call + +```mermaid +sequenceDiagram + participant App + participant Llm as Llm (llm.cpp) + participant Tok as Tokenizer+jinja + participant Emb as DiskEmbedding + participant Mod as Module pool (StaticModule/Session) + participant Pipe as Pipeline/Backend + participant Attn as CPUAttention+KVCacheManager + participant Gen as ArGeneration + participant Samp as Sampler + App->>Llm: response(ChatMessages, os, end_with, max_new) + Llm->>Tok: apply_chat_template(messages) -> prompt + Llm->>Llm: prompt_cache? diff vs mCachedPromptText -> delta tokens + Llm->>Tok: tokenizer_encode(prompt|delta) + Llm->>Llm: generate_init(os): reset ctx; if !reuse_kv: mMeta.remove=previous + Llm->>Emb: embedding(ids) -> VARP [seq,1,hidden] + Llm->>Llm: forwardVec: mMeta.add=seq; gen_attention_mask; gen_position_ids + Llm->>Mod: select key (100, all_logits) -> onForward({embeds,mask,pos,logits_index}) + Mod->>Pipe: Session resize (shape+geometry, exec cache) if dims changed; execute + Pipe->>Attn: OpType_Attention onExecute: onAlloc(KVMeta) grows KV by expandChunk; quant K/V; flash-attn blocks + Pipe-->>Mod: logits (last token) + Mod-->>Llm: outputs; mMeta.sync() (previous += add - remove) + Llm->>Gen: generate(max_new) [OpenCL: OP_ENCODER_NUMBER_FOR_COMMIT=512] + loop until stop token / max_new / USER_CANCEL / TIMEOUT + Gen->>Samp: sample(logits) -> token + Gen->>Llm: is_stop(token)? (sets NORMAL_FINISHED) + Gen->>Tok: tokenizer_decode(token) + Gen->>App: os << piece (flush) + Gen->>Llm: forwardVec({token}) -> module key (1,false), mMeta.add=1 + Llm->>Mod: onForward (no resize; KV append) + Mod-->>Gen: next logits + end + Gen-->>Llm: status MAX_TOKENS_FINISHED | NORMAL_FINISHED + Llm->>Llm: updateCachedPromptText (prompt_cache) + Llm-->>App: return (output_tokens in LlmContext) +``` + +### (b) TTS: Qwen2.5-Omni thinker → talker → DiT → BigVGAN (streaming) + +```mermaid +sequenceDiagram + participant App + participant Omni as Omni (omni.cpp) + participant Thinker as Llm modules (llm.mnn) + participant Talker as Talker (talker.mnn) + participant Q as WavChunk queues + participant DiT as ditWorkerLoop (predit.mnn + dit.mnn, own Executor) + participant Voc as vocoderWorkerLoop (bigvgan.mnn clone) + App->>Omni: setWavformCallback(cb) + App->>Omni: response(prompt, os) + Omni->>Talker: generate_init() + Omni->>Thinker: prefill (+vision/audio embeddings via visual.mnn/audio.mnn) + Thinker-->>Omni: logits + talker_embeds -> Talker.addTalkerEmbeds + loop thinker decode (text streamed to os) + Omni->>Thinker: forwardVec(token) + Thinker-->>Omni: talker_embeds[i] + end + App->>Omni: generateWavform() (or interleaved: talker steps inside the text loop) + Omni->>Talker: generate(): stepPrefill(concat(talker_embeds[0], text_bos+codec_pad, talker_embeds[1]+codec_bos)) + loop codec tokens (max talker_max_new_tokens) until 8292/8294 + Talker->>Talker: stepForward: embedding(prev)+talker_embeds[i]|text_pad -> forward -> sample (penalty,topK,topP,temp) + Talker->>Q: trySubmitChunkAsync: when >= 24+60+12 tokens push WavChunk(codec_tokens, noise slice) + end + Talker->>Q: finalize(): push last chunk, wait mWavLastDone (<=60s) + Q->>DiT: pop WavChunk + DiT->>DiT: mPreDit({cond,spk,code}) -> code_embeds,rope,mask; dit_steps-1 ODE steps (Euler or RK4) of mDit({x,code_embeds,rope,mask,t}) -> mel[80, 2*codec] + DiT->>Q: push chunk with mel (left/right padding sliced) + Q->>Voc: pop mel chunk; concat into mMelBuffer + Voc->>Voc: mBigvgan(mel) -> waveform; trim 8*240 samples context each side + Voc->>App: cb(pcm_ptr, n_samples, is_last) (return false => USER_CANCEL) + Omni-->>App: prints talker prefill/decode, token2wav time, tts rtf +``` + +[Verified] Qwen3-TTS variant (`Omni::generateTTS` → `Talker::generateQwen3TTS`, [omni.cpp#L3222](../../MNN/transformers/llm/engine/src/omni.cpp#L3222)): prompt embedder (codec prefix + text + tts tokens, optional ECAPA speaker embedding from `--ref_audio`) → per-frame loop: talker → first code (2048 + EOS) → `code_predictor` autoregressively over 16 groups → `codec_embedder` next input → after all frames one `speech_decoder({codes[1,16,frames]}) → waveform` → single callback with `last=true`. Not chunk-streamed. + +--- + +## 8. Tests and examples + +- [Verified] Unit tests: [test/](../../MNN/test) (`op/` 103 files, `expr/` 21, `speed/` 17, `grad/`, `core/`, `plugin/`, `model/`, `cv/`, `kleidiai/`, `sharedmem/`); runner `./run_test.out [test_name] [backend 0=CPU,3=OpenCL] [precision] [thread/gpuMode] [flag] [memory]` ([test/main.cpp](../../MNN/test/main.cpp)); `MNN_TEST_SKIP` env; attention/KV tests [AttentionTest.cpp](../../MNN/test/op/AttentionTest.cpp), GEMM/GEMV microbenchmarks [GemmSpeed.cpp](../../MNN/test/speed/GemmSpeed.cpp), [GemvBWTest.cpp](../../MNN/test/speed/GemvBWTest.cpp). +- [Verified] Driver [test.sh](../../MNN/test.sh) (`static | local | android [filter]`) with stages in [test_stages.json](../../MNN/test_stages.json) (`unit → lowmem → smokeA → smokeB → bench → llm`, filters `cpu|opencl-image|opencl-buffer|vulkan|lowmem|llm`, documented driver-bug skip rationales for Mali/OpenCL-image); [docs/testing.md](../../MNN/docs/testing.md); skill [skills/test-ci/SKILL.md](../../MNN/skills/test-ci/SKILL.md) incl. real-device iPhone LLM bench ([ios_llm_bench.sh](../../MNN/transformers/llm/engine/ios/ios_llm_bench.sh)). +- [Verified] Model/tool tests: [tools/cpp](../../MNN/tools/cpp) (`MNNV2Basic`, `ModuleBasic`, `backendTest`, `testModel`, `timeProfile`, `GetMNNInfo`, `MNN2QNNModel`, `compilefornpu`); converter round-trips [tools/script](../../MNN/tools/script) (`testMNNFromOnnx.py/Tf/Tflite/Torch`, `testPTQ.py`, `testTransformerC4Switches.py`); pymnn [pymnn/test](../../MNN/pymnn/test). +- [Verified] LLM: [engine/test](../../MNN/transformers/llm/engine/test) (`test_tokenizer`, `test_omni_video_prompt`, `test_multi_instance`), evals [transformers/llm/eval](../../MNN/transformers/llm/eval) (perplexity, C-Eval), [llm_bench.cpp](../../MNN/transformers/llm/engine/tools/llm_bench.cpp) (`-p 512` pp, `-n 128` tg, `-pg`), `bench_ttfa.py` (time to first audio), nightly [llm_nightly.sh](../../MNN/transformers/llm/benchmark/llm_nightly.sh). +- [Verified] CI ([.github/workflows](../../MNN/.github/workflows)): `linux.yml` builds 3 configs and runs `run_test.out` (incl. LLM/vision/audio/OpenCL/Vulkan-enabled build); `macos.yml`, `windows.yml`, `android.yml` (build-only), `ios.yml`, pymnn wheels, `code-format.yml`. **No on-device GPU/NPU CI in GitHub Actions.** +- [Verified] Examples: [demo/exec](../../MNN/demo/exec) (pictureRecognition, segment, multiPose, transformerDemo, expressDemo…), [benchmark/](../../MNN/benchmark) (`benchmark.out`, Android script), apps: [MnnLlmChat](../../MNN/apps/Android/MnnLlmChat) (LLM/VLM/audio-in/audio-out/diffusion, model market HF+ModelScope+Modelers), [MnnTaoAvatar](../../MNN/apps/Android/MnnTaoAvatar) (LLM+ASR+TTS+A2BS+NNR avatar), [iOS MNNLLMChat](../../MNN/apps/iOS/MNNLLMChat), [apps/sana](../../MNN/apps/sana), [apps/mnncli](../../MNN/apps/mnncli), TTS SDK demo [mnn_tts/demo/android](../../MNN/apps/frameworks/mnn_tts/demo/android). + +--- + +## 9. TTS relevance + +**What exists** [Verified] +- **Engine-native speech output** (`Talker : Llm`, [omni.hpp](../../MNN/transformers/llm/engine/src/omni.hpp)): + - Qwen2.5-Omni: `talker.mnn` (codec-token LM conditioned on thinker hidden states), `spk_dict.mnn` (speakers `Chelsie`, `Ethan`), `predit.mnn` + `dit.mnn` (flow-matching DiT, `dit_steps=5`, `dit_solver` 1 = Euler / 4 = RK4, forced `Precision_High` on GPU), `bigvgan.mnn` vocoder; 50 codec tokens ≈ 1 s (repo doc); chunked streaming (60-token DiT chunks, 24/12 context, vocoder 8/8 context at 240× upsample), async two-thread token2wav on CPU, interleaved thinker/talker mode (`interleaved=true`), RTF reporting. + - Qwen3-TTS (`talker_type=qwen3_tts`): talker + `code_predictor` (16 groups × 2048) + `codec_embedder` + `speech_decoder` (upsample 1920) + `speaker_encoder` (ECAPA-TDNN; **reference audio required** — voice cloning only); languages `auto/chinese/english/german/italian/portuguese/spanish/japanese/korean/french/russian`; API `Llm::generateTTS(text, language, max_new_tokens, ref_audio)`; demo [qwen3_tts_demo.cpp](../../MNN/transformers/llm/engine/demo/qwen3_tts_demo.cpp). + - Export: [utils/talker.py](../../MNN/transformers/llm/export/utils/talker.py) (`Qwen2_5OmniTalker`, `Qwen3TTSTalker`), [utils/token2wav.py](../../MNN/transformers/llm/export/utils/token2wav.py) (`DiTBlock`, `DiTAttention`, `DitRotary`, BigVGAN anti-alias surgery, `ECAPA_TDNN`, `Qwen2_5OmniToken2Wav`, `Qwen3TTSToken2Wav`). + - Config keys ([llmconfig.hpp#L201](../../MNN/transformers/llm/engine/src/llmconfig.hpp#L201)): `has_talker, talker_model/weight/embedding_file, talker_type, predit_model, dit_model, bigvgan_model, spk_dict, talker_speaker, talker_max_new_tokens, dit_steps, dit_solver, interleaved, code_predictor_*, speech_decoder_*, speaker_encoder_*, tts_{bos,eos,pad}_token_id`. + - App integration: Android `LlmSession_setWavformCallbackNative` writes `output.wav` @24 kHz and plays chunks ([llm_session.cpp](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp), `AudioChunksPlayer.kt`); iOS `setAudioWaveformCallback`; docs [llm.md](../../MNN/docs/transformers/llm.md) (AudioQueue playback example). +- **Native TTS SDK** ([apps/frameworks/mnn_tts](../../MNN/apps/frameworks/mnn_tts)): `MNNTTSSDK::Process(text) → (sample_rate, Audio)` with backends Bert-VITS2 (Chinese/English G2P, BERT front-ends, `TTSGenerator`), Supertonic (duration predictor / text encoder / vector estimator / vocoder, multi-voice styles), Piper (espeak-ng phonemizer; currently commented out); Android service + JNI, iOS, desktop. Shipped models `bert-vits2-MNN` (1.39 GB) and `supertonic-tts-mnn` (464 MB) in [model_market.json](../../MNN/apps/Android/MnnLlmChat/app/src/main/assets/model_market.json). +- **sherpa-mnn** ([apps/frameworks/sherpa-mnn](../../MNN/apps/frameworks/sherpa-mnn)): sherpa-onnx ported onto MNN: offline TTS VITS / Kokoro / Matcha + HiFiGAN, MeloTTS/Piper lexicons, streaming Zipformer ASR, VAD, speaker ID, C/Kotlin/Python APIs. Used by both Android apps for voice chat (ASR in, TTS out). +- **Audio input (ASR side)**: `audio.mnn` encoders for Qwen2-Audio, Qwen3-ASR (`qwen3_asr`), Qwen2.5-Omni audio, FunAudioChat, LFM2-Audio, Gemma4 audio ([utils/audio.py](../../MNN/transformers/llm/export/utils/audio.py)); `MNN::AUDIO` front-ends (whisper/conformer/usm); Whisper itself is not a shipped pipeline (only its fbank). +- **Ops**: `OpType_Stft` (shape/CPU kernel, [ShapeStft.cpp](../../MNN/source/shape/ShapeStft.cpp)); everything else (mel, iSTFT, vocoder convs/transposed convs/snake activations) is expressed as ordinary graph ops after export. + +**What's missing** [Verified unless noted] +- No CosyVoice / Fish-Speech / F5-TTS / SparkTTS / ChatTTS / Kokoro-native (only via sherpa) / SNAC / EnCodec / DAC / WavTokenizer; no Qwen3-Omni. +- Qwen3-TTS path is not chunk-streamed (single `speech_decoder` call) and has no preset speakers (ref audio mandatory). +- No sample-rate conversion on output; fixed 24 kHz assumption in demos. +- No engine-level audio backpressure/queue bounds; no `USER_CANCEL` for text other than via wav callback. +- Python binding lacks `setWavformCallback`/`generateTTS`. +- [Proposal] For vLLM-Omni-style stage separation, the natural seams are `Talker::stepForward` (codec LM) and the `WavChunk` queues (DiT / vocoder) — they are already separate Modules with their own RuntimeManager/Executor, so they could be placed on CPU vs GPU or different cores. + +--- + +## 10. VLA relevance + +**What exists** [Verified] +- Vision encoders exportable with dynamic resolution + M-RoPE: `Qwen2/2.5/3/3.5-VL`, `Qwen2.5-Omni`, `Qwen-VL`, `DeepSeek-VL`, `InternVL`, `GLM-OCR`, `Gemma3/4`, `Idefics3/SmolVLM`, `HunyuanVL`, `MiniCPM-V`, `MobileCLIP`, `LFM2-VL`, `FastVLM (llava_qwen2)` ([utils/vision.py](../../MNN/transformers/llm/export/utils/vision.py)); runtime preprocessing in `Omni::*VisionProcess` incl. video frame sampling; window attention masks; separate processor runtime (`mllm` config) so the encoder can run on another backend. +- Structured prompt input: `MultimodalPrompt{images (raw RGB buffers), audios (waveform), videos (frames+timestamps)}`; `input_embeds` overload of `response/generate` lets a host inject arbitrary continuous embeddings (e.g. proprioception projected on the host) [Verified API; use for proprio is a Proposal]. +- Diffusion/flow machinery: `DiT` blocks + host-driven ODE loop (`Talker::ditForward`), SD3.5/Sana flow matching, Wan2.1 T2V; generic `Module` API can run any exported action-head graph; `Session_Resize_Check/Fix` keeps repeated fixed-shape denoise steps allocation-free. +- Low-latency primitives: decode key-1 module (no resize), `reuse_kv`, `eraseHistory` (rollback), KV quant, fp16 ARM, KleidiAI/SME2, NPU delegation (QNN/HIAI/CoreML) for static-shape encoders. + +**What's missing** [Verified] +- Zero VLA/robotics code: no OpenVLA / π0 / SmolVLA / RDT / Octo / LeRobot, no action tokenizer/detokenizer (e.g. FAST), no action chunking, no proprio embedding path, no continuous-action head, no diffusion-policy scheduler, no control-loop/timer, no ROS/serial I/O. +- No multi-request batching (action chunk sampling with CFG would need batch-2 graphs exported explicitly). +- [Proposal] A VLA on MNN would be: export vision tower via `utils/vision.py`, export the LLM with `hidden_states=true` (already used by MTP/EAGLE) to expose the last hidden state, export the action head (MLP or DiT) as a separate `.mnn` Module, and drive the flow-matching loop on the host exactly as `Talker::ditForward` does. + +--- + +## 11. Edge deployment profile + +- **Build system** [Verified]: CMake ≥ 3.6, C++11 default (C++17 for CUDA+transformer-fuse, C++20 for MSVC+QNN), Google-style, `-fno-rtti -fno-exceptions`, `-fvisibility=hidden -ffunction-sections -fdata-sections`, `-O3` (no `-Os`), `--gc-sections -s` only for `MNN_BUILD_FOR_ANDROID_COMMAND` ([CMakeLists.txt](../../MNN/CMakeLists.txt)). Key options (defaults): `MNN_BUILD_SHARED_LIBS=ON`, `MNN_SEP_BUILD=ON` (separate `libMNN_CL/Vulkan/Express/OpenCV/Audio/llm/diffusion.so`), `MNN_USE_THREAD_POOL=ON`, `MNN_ARM82=ON`, `MNN_KLEIDIAI=ON` (`MNN_KLEIDIAI_DEFAULT_ON=OFF`), `MNN_SME2=ON`, `MNN_AVX2=ON`, `MNN_AVX512=OFF`, `MNN_USE_RVV=OFF`, `MNN_METAL/OPENCL/VULKAN/OPENGL/CUDA/MUSA/TENSORRT/COREML/NNAPI/QNN/RKNN/HEXAGON=OFF`, `MNN_LOW_MEMORY=OFF` (forced ON by `MNN_BUILD_LLM`), `MNN_SUPPORT_TRANSFORMER_FUSE=OFF` (forced ON by LLM/diffusion), `MNN_BUILD_LLM=OFF`, `MNN_BUILD_LLM_OMNI=OFF` (forces OpenCV+Audio+imgcodecs), `MNN_BUILD_DIFFUSION=OFF`, `MNN_BUILD_OPENCV=OFF`, `MNN_IMGCODECS=OFF`, `MNN_BUILD_AUDIO=OFF`, `MNN_WITH_PLUGIN=OFF`, `MNN_BUILD_MINI=OFF` (= `MNN_SKIPBUILD_GEOMETRY` + `MNN_REDUCE_SIZE`), `MNN_SUPPORT_BF16=OFF`, `MNN_USE_SYSTEM_LIB=OFF` (dlopen OpenCL/Vulkan), `MNN_JNI=OFF`, `MNN_BUILD_CONVERTER=OFF`, `MNN_BUILD_TEST=OFF`. `MNN_CPU_WEIGHT_DEQUANT_GEMM` was removed (comment in CMakeLists; superseded by `MNN_LOW_MEMORY`). Docs/code drift: [docs/compile/cmake.md](../../MNN/docs/compile/cmake.md) lists `MNN_USE_SSE` default OFF (code ON) and options no longer present (`MNN_METALLIB_SOURCE`, `MNN_VULKAN_DEBUG`). +- **Binary size** [repo-reported, not reproduced]: [README.md](../../MNN/README.md) — iOS full static lib armv7+arm64 ≈ 12 MB (+2 MB executable growth), Android core `.so` ≈ 800 KB (armv7a, `c++_shared`), `MNN_BUILD_MINI` ≈ −25%; [docs/intro/about.md](../../MNN/docs/intro/about.md) — trimmed iOS static ≈ 6.1 MB (+600 KB). Op pruning: `GetMNNInfo` → `tools/script/prue_mnn_ops.py` rewrites `CPUOPRegister.cpp`/`GeometryOPRegister.cpp` ([docs/faq.md](../../MNN/docs/faq.md)). +- **Runtime dependencies** [Verified]: minimal `libMNN`: Linux `-pthread -ldl`; Android `log m android`; OHOS `libhilog_ndk.z.so`; flatbuffers header-only vendored; protobuf converter-only; OpenGL is the only GPU backend hard-linked (`GLESv3`, `EGL`). Vendored [3rd_party](../../MNN/3rd_party): flatbuffers 1.10, half, stb (imageHelper), OpenCL headers, rapidjson, protobuf (converter), musa_compat. Fetched at build: KleidiAI v1.16.0, CUTLASS (CUDA), oneDNN (opt), googletest (cv/audio tests), QNN SDK zip ([prepare_qnn_deps.sh](../../MNN/prepare_qnn_deps.sh)). Additional LLM deps: cpp-httplib (vendored), jinja.cpp (vendored), OpenSSL only for `mls`. +- **Platforms** [Verified]: Android NDK arm64-v8a (API 21, [project/android/build_64.sh](../../MNN/project/android/build_64.sh): OpenCL + LLM + diffusion) and armeabi-v7a (API 14, [package_scripts/android/build.sh](../../MNN/package_scripts/android/build.sh)); iOS ≥ 11 via [MNN.podspec](../../MNN/MNN.podspec) / `MNN.framework` ([package_scripts/ios/buildiOS.sh](../../MNN/package_scripts/ios/buildiOS.sh), Metal + ARM82, thread pool off); macOS arm64+x86_64 fat framework; Linux x86/arm64 (cross-compile toolchain [project/cross-compile](../../MNN/project/cross-compile)); Windows MSVC/clang incl. ARM64; OpenHarmony ([project/harmony/build_64.sh](../../MNN/project/harmony/build_64.sh)); WASM via emcmake ([docs/compile/engine.md](../../MNN/docs/compile/engine.md), LLM-on-WASM recipe in llm.md); RISC-V RVV + SpacemiT K3 IME2 ([riscv/CMakeLists.txt](../../MNN/source/backend/cpu/riscv/CMakeLists.txt), perf note [riscv_k3_ime2_asymmetric_w4b64.md](../../MNN/docs/perf/riscv_k3_ime2_asymmetric_w4b64.md)); no LoongArch. +- **Accelerators** [Verified]: NVIDIA CUDA sm 60/61/62/70/72/75/80/86/**87**/89/120 ([cuda/CMakeLists.txt](../../MNN/source/backend/cuda/CMakeLists.txt); sm87 = Jetson Orin [Inferred], CUDA ≥ 11.4), TensorRT (`MNN_TRT_DYNAMIC` dlopen); Apple Metal (Metal4 tensor ops via `MNN_METAL_TENSOR`), CoreML/ANE (`MNN_FORWARD_NN`); Android OpenCL (dlopen search list for Mali/Adreno/PowerVR/pocl in [OpenCLWrapper.cpp](../../MNN/source/backend/opencl/core/runtime/OpenCLWrapper.cpp)), Vulkan (dlopen [vulkan_wrapper.cpp](../../MNN/source/backend/vulkan/runtime/vulkan_wrapper.cpp)), NNAPI (API ≥ 29), Qualcomm QNN/HTP (online: static-shape generic `.mnn`, `MNN_FORWARD_NN`; offline: `MNN2QNNModel` → `model__.bin` + plugin `.mnn` run on CPU type with `MNN_WITH_PLUGIN`; SoC table 8 Gen1/2/3/Elite = 36/69, 43/73, 57/75, 69/79 in [docs/inference/npu.md](../../MNN/docs/inference/npu.md); LLM-on-QNN export [llm.md](../../MNN/docs/transformers/llm.md)), Hexagon HVX direct (`libMNN_htpops*.so` prebuilt with Hexagon SDK, [hexagon/README.md](../../MNN/source/backend/hexagon/README.md)), Huawei HIAI (`MNN_NPU`, prebuilt DDK libs), Rockchip RKNN (`MNNConvert --rknn` → wrapper `.mnn` + `.rknn`, fp32 output only, host copies), MediaTek NeuroPilot not shipped; Moore Threads MUSA. +- **Model conversion path & constraints** [Verified]: generic CNN/transformer: `MNNConvert -f ONNX|TF|TFLITE|CAFFE|TORCH --modelFile … --MNNModel … [--fp16|--weightQuantBits N --weightQuantBlock B] [--saveExternalData] [--saveStaticModel]`; LLM: `llmexport.py --path --export mnn [--quant_bit 4 --quant_block 64 --hqq]`, needs `MNNConvert` (or pymnn) and produces fixed batch-1 graphs; fixed-shape backends (OpenCL/Vulkan/NPU) require `shapeMutable=false` + chunked prefill; QNN needs static shapes per (chunk, history) bucket; NPU export scripts are Qualcomm/MLDA specific; TTS needs `MNN_BUILD_AUDIO` (+ `LLM_SUPPORT_AUDIO` define) and vision needs `MNN_BUILD_OPENCV` (+`MNN_IMGCODECS`). Pre-converted models published on ModelScope `MNN` org and HF `taobao-mnn` ([docs/transformers/models.md](../../MNN/docs/transformers/models.md)). +- **Python wheel** [Verified]: `pip install MNN` (repo-reported); source build `pymnn/pip_package` with deps string incl. `llm`, `opencl`, `vulkan`, `cuda`, `torch` ([docs/compile/pymnn.md](../../MNN/docs/compile/pymnn.md)); console tools `mnnconvert`, `mnnquant`. + +--- + +## 12. Limitations and unknowns + +- [Verified] No request-level scheduler, batching, or paged KV: one request per `Llm`, batch dim 1 baked into exported graphs. +- [Verified] No engine API to cancel a running text `response()`; apps must step `generate(1)` themselves. `USER_CANCEL` only via wav callback. +- [Verified] KV cache is a contiguous growable array per attention execution; no eviction beyond explicit `remove/reserve`; disk spill only with `kvcache_mmap`/`KVCACHE_SIZE_LIMIT` (the LLM engine never sets the size limit hint). +- [Verified] Flash attention / KV quant knobs are CPU-first: Metal supports int8 KV + FA with head_dim ∈ {64,128,256}; OpenCL/Vulkan/CUDA ignore `attention_mode` (repo doc). TQ3/TQ4 KV quant is CPU-only. +- [Verified] OpenCL/Vulkan image mode lacks `LinearAttention` (falls back to CPU); Vulkan low-bit kernels exist only in the buffer variant. +- [Verified] W2/W3 weights need ARMv8.6 i8mm + fp16 hardware; W4 GEMV on Metal accepts only 2/3/4/8-bit. +- [Verified] `Llm::tuning` is skipped on OpenCL; `Llm::load` performs no warmup, so first-token latency includes shader compile / tuning unless a cache file exists. +- [Verified] Talker async token2wav only on CPU backend; DiT forced to fp32 on GPU (fp16 collapses the ODE). +- [Verified] Qwen3-TTS requires reference audio (no preset speakers) and produces audio only after all frames; Qwen2.5-Omni speakers limited to Chelsie/Ethan. +- [Verified] No Whisper/SenseVoice/Paraformer pipeline in the engine (ASR delegated to sherpa-mnn Zipformer or Qwen-audio encoders). +- [Verified] No VLA/robotics code; no action tokenizer; no proprio path. +- [Verified] NeuroPilot backend is a stub in OSS; HIAI/NPU option undeclared (`-DMNN_NPU=ON` by hand); NNAPI and CoreML cannot coexist (same forward type). +- [Verified] No JIT/graph capture; GPU op launch overhead mitigated only by commit batching / record queues. +- [Verified] Python binding has no TTS/audio-out API; no C API; no WASM binding (build recipe only); no on-device CI in GitHub Actions. +- [Verified] Docs drift: cmake.md defaults vs code (`MNN_USE_SSE`, `MNN_AVX512_VNNI`), stale option names; `docs/inference/npu.md` omits Hexagon/NeuroPilot. +- [Unknown] Actual latency/RTF numbers on target edge devices — only repo-reported figures exist (e.g. Metal prefill +3.6–3.9% note [metal_prefill_optimization_summary.md](../../MNN/docs/perf/metal_prefill_optimization_summary.md); MnnLlmChat README speed-up claims vs llama.cpp/fastllm); none reproduced here. +- [Unknown] Behaviour of `schema/private/` / `source/internal/` features (model auth, metrics under `MNN_INTERNAL`) — not read per AGENTS.md. +- [Unknown] Whether the bounded-thread `ThreadPool` (`MNN_THREAD_POOL_MAX_TASKS=2`) limits running vision encoder and talker concurrently on the same runtime — the Talker avoids it by creating a fresh `Executor` per worker thread. +- [Inferred] Multi-model pipelines (encoder on NPU, LLM on CPU, vocoder on GPU) are feasible today via per-Module `RuntimeManager`s, but every cross-backend hop is a synchronous host copy (`onSync` + `copyToHostTensor`). + +--- + +## 13. Reference index + +| Path | Role | +|---|---| +| [../../MNN/AGENTS.md](../../MNN/AGENTS.md) | Repo guidance; restricted dirs; architecture summary | +| [../../MNN/README.md](../../MNN/README.md) | Overview; repo-reported size numbers; news (Qwen3.5, Qwen3-VL, Omni) | +| [../../MNN/CMakeLists.txt](../../MNN/CMakeLists.txt) | All top-level build options, platform flags, link deps | +| [../../MNN/MNN.podspec](../../MNN/MNN.podspec) | CocoaPods spec (iOS 11+) | +| [../../MNN/prepare_qnn_deps.sh](../../MNN/prepare_qnn_deps.sh) | Downloads QNN SDK headers/libs | +| [../../MNN/test.sh](../../MNN/test.sh), [../../MNN/test_stages.json](../../MNN/test_stages.json) | CI/test driver and stage matrix | +| [../../MNN/include/MNN/MNNDefine.h](../../MNN/include/MNN/MNNDefine.h) | Version 3.6.1 | +| [../../MNN/include/MNN/MNNForwardType.h](../../MNN/include/MNN/MNNForwardType.h) | Forward types, GPU mode flags, BackendConfig, RuntimeStatus | +| [../../MNN/include/MNN/Interpreter.hpp](../../MNN/include/MNN/Interpreter.hpp) | Session API, SessionMode, HintMode, ExternalPathType | +| [../../MNN/include/MNN/Tensor.hpp](../../MNN/include/MNN/Tensor.hpp) | Tensor, DimensionType, map/wait | +| [../../MNN/include/MNN/ImageProcess.hpp](../../MNN/include/MNN/ImageProcess.hpp) | Image preprocessing API | +| [../../MNN/include/MNN/MNNSharedContext.h](../../MNN/include/MNN/MNNSharedContext.h) | Only `extern "C"` header (shared GPU context) | +| [../../MNN/include/MNN/expr/Module.hpp](../../MNN/include/MNN/expr/Module.hpp) | Module::load/Config/clone | +| [../../MNN/include/MNN/expr/Executor.hpp](../../MNN/include/MNN/expr/Executor.hpp) | Executor, RuntimeManager, LazyMode | +| [../../MNN/include/MNN/plugin](../../MNN/include/MNN/plugin) | Plugin op kernel/shape registration | +| [../../MNN/source/core/Backend.hpp](../../MNN/source/core/Backend.hpp), [Backend.cpp](../../MNN/source/core/Backend.cpp) | Backend/Runtime/RuntimeHint; creator registry | +| [../../MNN/source/core/RuntimeFactory.cpp](../../MNN/source/core/RuntimeFactory.cpp) | Runtime creation | +| [../../MNN/source/core/Schedule.cpp](../../MNN/source/core/Schedule.cpp) | Backend type resolution, op scheduling | +| [../../MNN/source/core/Session.cpp](../../MNN/source/core/Session.cpp) | Session lifecycle, hint translation, backup backend | +| [../../MNN/source/core/Pipeline.cpp](../../MNN/source/core/Pipeline.cpp) | encode/allocMemory/execute, CPU fallback, copies, tuning | +| [../../MNN/source/core/WrapExecution.cpp](../../MNN/source/core/WrapExecution.cpp) | Cross-backend/layout copy insertion | +| [../../MNN/source/core/Execution.hpp](../../MNN/source/core/Execution.hpp) | Execution interface | +| [../../MNN/source/core/BufferAllocator.hpp](../../MNN/source/core/BufferAllocator.hpp) | Eager/Defer allocators, mmap allocator | +| [../../MNN/source/core/MNNMemoryUtils.h](../../MNN/source/core/MNNMemoryUtils.h) | 64-byte alignment | +| [../../MNN/source/core/Tensor.cpp](../../MNN/source/core/Tensor.cpp) | Host/device copy and map | +| [../../MNN/source/core/TensorUtils.hpp](../../MNN/source/core/TensorUtils.hpp) | NC4HW4 descriptor, Region | +| [../../MNN/source/core/ConvolutionCommon.cpp](../../MNN/source/core/ConvolutionCommon.cpp) | Quantized weight decode | +| [../../MNN/source/core/KVMeta.hpp](../../MNN/source/core/KVMeta.hpp), [KVCacheManager.hpp](../../MNN/source/core/KVCacheManager.hpp) | KV cursor and storage base | +| [../../MNN/source/geometry/GeometryComputer.hpp](../../MNN/source/geometry/GeometryComputer.hpp), [GeometryComputer.cpp](../../MNN/source/geometry/GeometryComputer.cpp), [GeometryOPRegister.cpp](../../MNN/source/geometry/GeometryOPRegister.cpp) | Geometry decomposition | +| [../../MNN/source/shape/SizeComputer.hpp](../../MNN/source/shape/SizeComputer.hpp), [ShapeRegister.cpp](../../MNN/source/shape/ShapeRegister.cpp), [ShapeStft.cpp](../../MNN/source/shape/ShapeStft.cpp) | Shape inference | +| [../../MNN/source/backend/cpu/CPUBackend.cpp](../../MNN/source/backend/cpu/CPUBackend.cpp), [CPUOPRegister.cpp](../../MNN/source/backend/cpu/CPUOPRegister.cpp), [CPURuntime.cpp](../../MNN/source/backend/cpu/CPURuntime.cpp), [ThreadPool.cpp](../../MNN/source/backend/cpu/ThreadPool.cpp) | CPU backend, dispatch, core detection, thread pool | +| [../../MNN/source/backend/cpu/CPUAttention.cpp](../../MNN/source/backend/cpu/CPUAttention.cpp), [CPUKVCacheManager.hpp](../../MNN/source/backend/cpu/CPUKVCacheManager.hpp), [CPUKVCacheManager.cpp](../../MNN/source/backend/cpu/CPUKVCacheManager.cpp) | Attention, KV quant, disk KV | +| [../../MNN/source/backend/cpu/CPUPlugin.cpp](../../MNN/source/backend/cpu/CPUPlugin.cpp) | Plugin op execution | +| [../../MNN/source/backend/cpu/compute/CommonOptFunction.h](../../MNN/source/backend/cpu/compute/CommonOptFunction.h) | CoreFunctions table (pack/bytes/kernels) | +| [../../MNN/source/backend/cpu/compute/ConvolutionFloatFactory.cpp](../../MNN/source/backend/cpu/compute/ConvolutionFloatFactory.cpp) | Low-memory conv/GEMM selection | +| [../../MNN/source/backend/cpu/compute/ImageProcessFunction.cpp](../../MNN/source/backend/cpu/compute/ImageProcessFunction.cpp) | Image preprocessing kernels | +| [../../MNN/source/backend/cpu/arm/arm64/low_memory](../../MNN/source/backend/cpu/arm/arm64/low_memory) | W2/W3/W4 GEMM and dynamic quant asm | +| [../../MNN/source/backend/cpu/riscv/CMakeLists.txt](../../MNN/source/backend/cpu/riscv/CMakeLists.txt) | RISC-V RVV / SpacemiT build | +| [../../MNN/source/backend/arm82/Arm82Backend.hpp](../../MNN/source/backend/arm82/Arm82Backend.hpp) | fp16 backend (NC8HW8) | +| [../../MNN/source/backend/metal/MetalBackend.mm](../../MNN/source/backend/metal/MetalBackend.mm), [MetalOPRegister.mm](../../MNN/source/backend/metal/MetalOPRegister.mm), [MetalAttention.mm](../../MNN/source/backend/metal/MetalAttention.mm), [AllShader.cpp](../../MNN/source/backend/metal/AllShader.cpp), [MetalCache.fbs](../../MNN/source/backend/metal/schema/MetalCache.fbs) | Metal backend | +| [../../MNN/source/backend/opencl/core/OpenCLBackend.cpp](../../MNN/source/backend/opencl/core/OpenCLBackend.cpp), [OpenCLOPRegister.cpp](../../MNN/source/backend/opencl/core/OpenCLOPRegister.cpp), [OpenCLWrapper.cpp](../../MNN/source/backend/opencl/core/runtime/OpenCLWrapper.cpp), [AttentionBufExecution.cpp](../../MNN/source/backend/opencl/execution/buffer/AttentionBufExecution.cpp) | OpenCL backend | +| [../../MNN/source/backend/vulkan/runtime/VulkanRuntime.cpp](../../MNN/source/backend/vulkan/runtime/VulkanRuntime.cpp), [vulkan_wrapper.cpp](../../MNN/source/backend/vulkan/runtime/vulkan_wrapper.cpp), [buffer/backend/VulkanBackend.cpp](../../MNN/source/backend/vulkan/buffer/backend/VulkanBackend.cpp), [VulkanAttention.cpp](../../MNN/source/backend/vulkan/buffer/execution/VulkanAttention.cpp), [buffer/compiler](../../MNN/source/backend/vulkan/buffer/compiler) | Vulkan backend | +| [../../MNN/source/backend/cuda/Register.cpp](../../MNN/source/backend/cuda/Register.cpp), [cuda/CMakeLists.txt](../../MNN/source/backend/cuda/CMakeLists.txt), [AttentionExecution.cu](../../MNN/source/backend/cuda/execution/AttentionExecution.cu) | CUDA backend | +| [../../MNN/source/backend/opengl/GLBackend.cpp](../../MNN/source/backend/opengl/GLBackend.cpp) | OpenGL backend | +| [../../MNN/source/backend/coreml](../../MNN/source/backend/coreml), [nnapi](../../MNN/source/backend/nnapi), [qnn](../../MNN/source/backend/qnn), [hiai](../../MNN/source/backend/hiai), [hexagon](../../MNN/source/backend/hexagon), [hexagon/README.md](../../MNN/source/backend/hexagon/README.md), [rknn](../../MNN/source/backend/rknn), [neuropilot/CMakeLists.txt](../../MNN/source/backend/neuropilot/CMakeLists.txt), [musa](../../MNN/source/backend/musa), [tensorrt](../../MNN/source/backend/tensorrt) | NPU / other accelerator backends | +| [../../MNN/express/Executor.cpp](../../MNN/express/Executor.cpp), [Utils.hpp](../../MNN/express/Utils.hpp), [module/Module.cpp](../../MNN/express/module/Module.cpp), [module/StaticModule.cpp](../../MNN/express/module/StaticModule.cpp), [module/PipelineModule.hpp](../../MNN/express/module/PipelineModule.hpp) | Express runtime and Module implementations | +| [../../MNN/schema/default/MNN.fbs](../../MNN/schema/default/MNN.fbs), [CaffeOp.fbs](../../MNN/schema/default/CaffeOp.fbs), [TensorflowOp.fbs](../../MNN/schema/default/TensorflowOp.fbs), [Tensor.fbs](../../MNN/schema/default/Tensor.fbs), [UserDefine.fbs](../../MNN/schema/default/UserDefine.fbs), [TFQuantizeOp.fbs](../../MNN/schema/default/TFQuantizeOp.fbs), [Type.fbs](../../MNN/schema/default/Type.fbs), [ExtraInfo.fbs](../../MNN/schema/default/ExtraInfo.fbs), [TrainInfo.fbs](../../MNN/schema/default/TrainInfo.fbs) | Model schema | +| [../../MNN/codegen/CMakeLists.txt](../../MNN/codegen/CMakeLists.txt) | Source-level fusion codegen options | +| [../../MNN/cmake/KleidiAI.cmake](../../MNN/cmake/KleidiAI.cmake) | KleidiAI fetch/integration | +| [../../MNN/tools/converter/source/MNNConverter.cpp](../../MNN/tools/converter/source/MNNConverter.cpp), [common/cli.cpp](../../MNN/tools/converter/source/common/cli.cpp), [common/writeFb.cpp](../../MNN/tools/converter/source/common/writeFb.cpp), [include/config.hpp](../../MNN/tools/converter/include/config.hpp), [optimizer/PostConverter.cpp](../../MNN/tools/converter/source/optimizer/PostConverter.cpp), [optimizer/merge](../../MNN/tools/converter/source/optimizer/merge), [postconvert/FuseTransformerC4.cpp](../../MNN/tools/converter/source/optimizer/postconvert/FuseTransformerC4.cpp) | Converter | +| [../../MNN/tools/quantization/calibration.cpp](../../MNN/tools/quantization/calibration.cpp) | Offline int8 calibration | +| [../../MNN/tools/cpp](../../MNN/tools/cpp), [compilefornpu.cpp](../../MNN/tools/cpp/compilefornpu.cpp) | Test/benchmark/NPU-compile tools | +| [../../MNN/tools/script](../../MNN/tools/script) | Converter/model test scripts, op pruning | +| [../../MNN/tools/cv/include/cv/cv.hpp](../../MNN/tools/cv/include/cv/cv.hpp) | OpenCV-like API | +| [../../MNN/tools/audio/include/audio/audio.hpp](../../MNN/tools/audio/include/audio/audio.hpp) | Audio front-end API | +| [../../MNN/transformers/llm/export/llmexport.py](../../MNN/transformers/llm/export/llmexport.py), [gguf2mnn.py](../../MNN/transformers/llm/export/gguf2mnn.py), [safetensors2mnn.py](../../MNN/transformers/llm/export/safetensors2mnn.py), [utils/model_mapper.py](../../MNN/transformers/llm/export/utils/model_mapper.py), [utils/vision.py](../../MNN/transformers/llm/export/utils/vision.py), [utils/audio.py](../../MNN/transformers/llm/export/utils/audio.py), [utils/talker.py](../../MNN/transformers/llm/export/utils/talker.py), [utils/token2wav.py](../../MNN/transformers/llm/export/utils/token2wav.py), [utils/mnn_converter.py](../../MNN/transformers/llm/export/utils/mnn_converter.py), [utils/mnn_utils.py](../../MNN/transformers/llm/export/utils/mnn_utils.py), [export/npu](../../MNN/transformers/llm/export/npu) | LLM/VLM/TTS export | +| [../../MNN/transformers/llm/engine/include/llm/llm.hpp](../../MNN/transformers/llm/engine/include/llm/llm.hpp), [reranker.hpp](../../MNN/transformers/llm/engine/include/llm/reranker.hpp) | Public LLM API | +| [../../MNN/transformers/llm/engine/src/llm.cpp](../../MNN/transformers/llm/engine/src/llm.cpp), [llmconfig.hpp](../../MNN/transformers/llm/engine/src/llmconfig.hpp), [omni.hpp](../../MNN/transformers/llm/engine/src/omni.hpp), [omni.cpp](../../MNN/transformers/llm/engine/src/omni.cpp), [embedding.cpp](../../MNN/transformers/llm/engine/src/embedding.cpp), [diskembedding.cpp](../../MNN/transformers/llm/engine/src/diskembedding.cpp), [sampler.cpp](../../MNN/transformers/llm/engine/src/sampler.cpp), [kvmeta.hpp](../../MNN/transformers/llm/engine/src/kvmeta.hpp), [prompt_cache_utils.hpp](../../MNN/transformers/llm/engine/src/prompt_cache_utils.hpp) | LLM engine core | +| [../../MNN/transformers/llm/engine/src/speculative_decoding](../../MNN/transformers/llm/engine/src/speculative_decoding), [generate.cpp](../../MNN/transformers/llm/engine/src/speculative_decoding/generate.cpp) | AR / lookahead / MTP / EAGLE / DFlash generation | +| [../../MNN/transformers/llm/engine/src/tokenizer/tokenizer.cpp](../../MNN/transformers/llm/engine/src/tokenizer/tokenizer.cpp), [jinja.hpp](../../MNN/transformers/llm/engine/src/tokenizer/jinja.hpp) | Tokenizers, chat templates | +| [../../MNN/transformers/llm/engine/demo](../../MNN/transformers/llm/engine/demo), [qwen3_tts_demo.cpp](../../MNN/transformers/llm/engine/demo/qwen3_tts_demo.cpp) | CLI demos | +| [../../MNN/transformers/llm/engine/app/mls_server.cpp](../../MNN/transformers/llm/engine/app/mls_server.cpp) | Minimal OpenAI-style HTTP server | +| [../../MNN/transformers/llm/engine/tools/llm_bench.cpp](../../MNN/transformers/llm/engine/tools/llm_bench.cpp), [engine/test](../../MNN/transformers/llm/engine/test), [test_multi_instance.cpp](../../MNN/transformers/llm/engine/test/test_multi_instance.cpp), [engine/ios](../../MNN/transformers/llm/engine/ios), [ios_llm_bench.sh](../../MNN/transformers/llm/engine/ios/ios_llm_bench.sh) | LLM bench/tests/iOS demo | +| [../../MNN/transformers/llm/eval](../../MNN/transformers/llm/eval), [benchmark/llm_nightly.sh](../../MNN/transformers/llm/benchmark/llm_nightly.sh) | Evals and nightly | +| [../../MNN/transformers/diffusion/engine/include/diffusion/diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/diffusion.hpp), [sana_diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/sana_diffusion.hpp), [wan_diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/wan_diffusion.hpp), [src/scheduler.hpp](../../MNN/transformers/diffusion/engine/src/scheduler.hpp), [wan_diffusion_demo.cpp](../../MNN/transformers/diffusion/engine/wan_diffusion_demo.cpp), [diffusion/export](../../MNN/transformers/diffusion/export) | Diffusion engine | +| [../../MNN/apps/Android/MnnLlmChat](../../MNN/apps/Android/MnnLlmChat), [llm_session.cpp](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp), [llm_stream_buffer.hpp](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_stream_buffer.hpp), [model_market.json](../../MNN/apps/Android/MnnLlmChat/app/src/main/assets/model_market.json) | Android chat app | +| [../../MNN/apps/Android/MnnTaoAvatar](../../MNN/apps/Android/MnnTaoAvatar) | Avatar app (LLM+ASR+TTS+A2BS+NNR) | +| [../../MNN/apps/iOS/MNNLLMChat](../../MNN/apps/iOS/MNNLLMChat), [LLMInferenceEngineWrapper.h](../../MNN/apps/iOS/MNNLLMChat/MNNLLMiOS/InferenceEngine/LLMInferenceEngineWrapper.h) | iOS chat app | +| [../../MNN/apps/frameworks/mnn_tts](../../MNN/apps/frameworks/mnn_tts), [mnn_tts/demo/android](../../MNN/apps/frameworks/mnn_tts/demo/android) | Native TTS SDK (Bert-VITS2 / Supertonic / Piper) | +| [../../MNN/apps/frameworks/sherpa-mnn](../../MNN/apps/frameworks/sherpa-mnn) | sherpa-onnx port (TTS/ASR/VAD) | +| [../../MNN/apps/mnncli](../../MNN/apps/mnncli), [mnncli_server.cpp](../../MNN/apps/mnncli/src/mnncli_server.cpp) | CLI + OpenAI/Anthropic-compatible server | +| [../../MNN/apps/sana](../../MNN/apps/sana) | Sana image-edit app | +| [../../MNN/pymnn/src/llm.h](../../MNN/pymnn/src/llm.h), [pymnn/src/audio.h](../../MNN/pymnn/src/audio.h), [pymnn/test](../../MNN/pymnn/test) | Python bindings | +| [../../MNN/project/android/build_64.sh](../../MNN/project/android/build_64.sh), [project/harmony/build_64.sh](../../MNN/project/harmony/build_64.sh), [project/cross-compile](../../MNN/project/cross-compile), [package_scripts/android/build.sh](../../MNN/package_scripts/android/build.sh), [package_scripts/ios/buildiOS.sh](../../MNN/package_scripts/ios/buildiOS.sh) | Platform build/packaging scripts | +| [../../MNN/demo/exec](../../MNN/demo/exec), [benchmark](../../MNN/benchmark) | Classic demos and benchmark | +| [../../MNN/test](../../MNN/test), [test/main.cpp](../../MNN/test/main.cpp), [test/op/AttentionTest.cpp](../../MNN/test/op/AttentionTest.cpp), [test/speed/GemmSpeed.cpp](../../MNN/test/speed/GemmSpeed.cpp), [test/speed/GemvBWTest.cpp](../../MNN/test/speed/GemvBWTest.cpp) | Unit tests | +| [../../MNN/.github/workflows](../../MNN/.github/workflows) | CI | +| [../../MNN/skills/test-ci/SKILL.md](../../MNN/skills/test-ci/SKILL.md) | Test/CI agent skill | +| [../../MNN/docs/contribute/op.md](../../MNN/docs/contribute/op.md), [docs/contribute/backend.md](../../MNN/docs/contribute/backend.md) | Extension recipes | +| [../../MNN/docs/transformers/llm.md](../../MNN/docs/transformers/llm.md), [models.md](../../MNN/docs/transformers/models.md) | LLM export/config/backends/TTS docs; model hubs | +| [../../MNN/docs/inference/npu.md](../../MNN/docs/inference/npu.md) | NPU backends (QNN/CoreML/NNAPI/HIAI/RKNN) | +| [../../MNN/docs/compile/cmake.md](../../MNN/docs/compile/cmake.md), [engine.md](../../MNN/docs/compile/engine.md), [pymnn.md](../../MNN/docs/compile/pymnn.md) | Build docs | +| [../../MNN/docs/intro/about.md](../../MNN/docs/intro/about.md), [docs/faq.md](../../MNN/docs/faq.md), [docs/testing.md](../../MNN/docs/testing.md) | Size claims, pruning, testing | +| [../../MNN/docs/perf/metal_prefill_optimization_summary.md](../../MNN/docs/perf/metal_prefill_optimization_summary.md), [riscv_k3_ime2_asymmetric_w4b64.md](../../MNN/docs/perf/riscv_k3_ime2_asymmetric_w4b64.md) | Repo-reported perf notes | +| [../../MNN/3rd_party](../../MNN/3rd_party) | Vendored deps | diff --git a/docs/engines/mobile_htp_simulation.md b/docs/engines/mobile_htp_simulation.md new file mode 100644 index 000000000..ba866d30b --- /dev/null +++ b/docs/engines/mobile_htp_simulation.md @@ -0,0 +1,220 @@ +# Mobile NPU simulation with the Qualcomm AI Runtime SDK (Hexagon HTP), and the CPU speed-ups it triggered + +Goal set on 2026-09-10: install Qualcomm's AI Runtime SDK with its Hexagon HTP (NPU) simulator, use it to stand in for mobile devices, and make the branch beat the other engines on CPU and on mobile. This page records what was installed, what runs on the simulator, what the simulator can and cannot tell us, and the CPU changes measured on the way. Raw artifacts: `tools/qairt/` (SDK, runtime libs, Python env notes), `analysis/experiments/edge_engine_compare/results/cpu_opt/` (CPU runs), the session scratch directory for ONNX/DLC files (not committed, 460 MB each). + +## 1. What was installed and how + +| Item | Where | Notes | +|---|---|---| +| Qualcomm AI Runtime **Community** SDK 2.37.0.250724 (formerly QNN / AI Engine Direct) | `tools/qairt/qairt/2.37.0.250724/` | Downloaded from Qualcomm's software center without a login (the URL ExecuTorch's `backends/qualcomm/scripts/qnn_config.sh` pins); 1.35 GB zip. Contains the converters (`qairt-converter`, `qairt-quantizer`), `qnn-net-run`, `qnn-context-binary-generator`, profiling readers, and the backends for x86 (`libQnnCpu.so`, `libQnnHtp.so` = **HTP functional simulator + offline graph preparation**, `libQnnHtpQemu.so` = HTP QEMU backend, id 13), Android aarch64 and Hexagon v66–v79 skels. | +| LLVM runtime libraries the SDK binaries link against | `tools/qairt/runtime_libs/` (`libc++.so.1`, `libc++abi.so.1`, `libunwind.so.1`) | Not on this host and no root: unpacked from Ubuntu jammy `libc++1-14`, `libc++abi1-14` and `libunwind-18` `.deb` files with `dpkg-deb -x`; used via `LD_LIBRARY_PATH`. | +| Python 3.10 environment for the SDK tools | `.venvs/qairt` (uv) | The SDK supports 3.8/3.10 only. Its `check-python-dependency` installer needs `setuptools<81` (for `pkg_resources`) and does **not** install `onnx`; `onnx==1.16.2`, `onnxruntime==1.18.1`, `protobuf<5` were added. | +| ONNX exporter dependency | `onnxscript` in the CUDA venv | The TorchScript exporter fails on `aten::diff` in the decoder; the dynamo exporter (`torch.onnx.export(..., dynamo=True)`) works. | +| Qualcomm AI Hub | not installed | It is a cloud service (real devices behind an API token), not a local simulator; nothing on this host can stand in for it. | + +Traps hit (all fixed in the scripts under `analysis/experiments/edge_engine_compare/` and `tools/qairt/`): converter rejects ONNX `Reshape` with `allowzero=1` (dynamo emits it; cleared), rejects the `IsNaN`→`Where` guards the dynamo exporter puts after every softmax (stripped), and the HTP graph preparation fails on an **int Gather from a float table** (`node_embedding`, the RVQ codebook lookup) with `could not create op: q::Gather`, so the codebook lookup was moved to the host and the HTP graph takes the summed codebook embeddings as a float input. + +## 2. What runs on the HTP simulator + +Pipeline (scripts: `scratchpad/qnn/export_htp_decoder.py`, `htp_pipeline.sh`, `run_qnn.sh`): + +1. `WindowedCode2Wav` (branch `vllm_omni/edge/decoder_export.py`, context 72 frames + chunk 25) → `HtpBody` variant whose input is `quantized [1, 512, 97]` (RVQ sum done on the host) → ONNX (dynamo, opset 18, 458 MB fp32). +2. `qairt-converter --float_bitwidth 16` → DLC (fp16), 229 Reshape / 62 FullyConnected / 55 StridedSlice / 45 Transpose / 17 RmsNorm / 16 MatMul / 31 Conv2d / 6 ConvTranspose / 8 Softmax / ElementWiseNeuron (SnakeBeta) — every op accepted by the converter. +3. `qnn-net-run --backend libQnnHtp.so` on x86 with the HTP backend extensions (`dsp_arch v75`, `O=3`, `vtcm_mb 8`): graph preparation for HTP v75 **succeeds** (RmsNorm and Gelu were flagged "no properties registered" during prepare but did not stop it), execution runs on the functional simulator. + +Parity: the same DLC on the QNN **CPU** backend (fp32) reproduces the torch decoder bit-for-bit (SNR 107.6 dB, max |diff| 7e-6 over 48,000 samples) — the conversion is correct. The HTP fp16 execution of the 2-second decoder chunk was still running on the functional simulator when this document was written (3 h of wall time, 5 cores): the chunk is 481 GFLOP and the simulator executed the 2.67 GFLOP predictor body at ≈16 MFLOP/s, so the run should finish ≈8.5 h after its 01:38 start. A background chain runs `parity_dec_htp.py` against the torch reference when it exits and writes `analysis/experiments/edge_engine_compare/qnn/run/decoder_htp_parity.txt` (plus the profiling records). The same graph on the QEMU backend is not possible on this host (below). + +**Code predictor on the HTP simulator (real weights, complete).** Because the 15-pass code predictor is the CPU bottleneck (§3), it is the component that matters most on a phone NPU. Its 5-layer body (all 56 tensors loaded from the checkpoint, fused qkv/gate-up) was exported the same way (`export_predictor_htp.py`, input `x [1, 17, 1024]`), converted to fp32 and fp16 DLC (`Mul 82 / Add 46 / Reshape 35 / MatMul …`, all accepted), and run: + +| backend | result | +|---|---| +| QNN CPU backend, fp32 DLC | SNR **111.2 dB** vs torch, 41 ms per inference on 16 host threads (reference kernels) | +| **HTP v75 functional simulator, fp16 DLC** | graph preparation 2.3 s, execution completes; SNR **50.7 dB** / corr 0.999996 vs the fp32 torch body, i.e. fp16 HTP math is faithful for the predictor | +| HTP profiler (`--profiling_level detailed`, HTP profiling reader) | per-op records are produced (e.g. `rms_norm_0`, `node_linear` …) but their "cycles" are simulator cycles (3.3e11 for one pass), not Hexagon device cycles | + +**HTP QEMU backend (`libQnnHtpQemu.so`, backend id 13).** Tried for a second, closer-to-silicon data point. It refuses the HTP backend-extension config (`Unable to find HTP interface provider`) and without it fails at context creation (`err 0x3e8`); its strings explain why: it only loads *offline-prepared* context binaries and launches a Hexagon emulator subprocess whose command line must come from `QNN_HTP_QEMU_ARGS`, i.e. it needs the `qemu-hexagon` binary and the Hexagon skel/`libQnnHtpV75Skel.so` from the **Hexagon SDK**, which is a separate Qualcomm Package Manager download (account-gated) and not part of the community AI Runtime SDK. So on this host the HTP path is validated functionally (prepare + execute on the x86 simulator), not cycle-accurately. + +Artifacts: `analysis/experiments/edge_engine_compare/qnn/` (scripts, converter logs, run logs, profiling records, execution metadata); the ONNX/DLC files (150–460 MB) stay in the session scratch directory. + +What the simulator does and does not give: +- It is a **functional** simulator ("x86 Linux host backend serves as a functional simulator for the hardware accelerator and supports graph preparation", SDK backend docs): op support, graph preparation for a chosen Hexagon arch, fp16/int8 numerics, and the context binary that would be deployed. Wall time on x86 is not device time (one 2 s chunk takes minutes here). +- Per-op **cycle counts** come from the HTP profiler only when running on the device (the docs' detailed profiling "provides per op profiling result by cycle counts"); on the x86 simulator the profiling reader returns the host-side timeline. A device-time number therefore still needs a phone or Qualcomm AI Hub. The QEMU backend (`libQnnHtpQemu.so`) is exercised the same way (see below). + +## 3. CPU: making the branch faster than llama.cpp on the same host + +Starting point (§5 of [edge_device_simulation.md](edge_device_simulation.md)): 134 ms per 80 ms frame at 12+4 threads, of which 97 ms were the fifteen code-predictor sub-steps and 10 ms the sampler; llama.cpp's native path costs 85–97 ms per frame plus a 55 ms/frame serial vocoder (RTF 1.9–2.0). + +What was found and changed (branch commit `6ce8de1b`): + +| finding | change | effect (0.6B, 12+4 threads, 12 prompts) | +|---|---|---| +| each of the 15 sub-steps re-ran the 5-layer predictor over the whole prefix (up to 17 positions), no KV cache | `forward_cached`: per-layer KV cache, 2-token prefill then one token per sub-step; default on CPU | sub-step 6.4 → 4.7 ms; frame 134 → 108 ms; RTF 1.7 → 1.38 | +| the sub-step is per-operator overhead, not bandwidth (1.6 ms floor); OpenMP thread count irrelevant (4/2 threads: no change); per-submodule `torch.compile` worse (+2 ms, guard overhead per call) | shape-static `forward_static_step` (tensor position, boolean mask) compiled as one graph | micro-benchmark 70 → 58 ms per frame for the loop; engine run: see table below | +| bf16 M=1 GEMV kernels carry ~30 µs fixed overhead each; int8-dynamic Linear has none and halves the bytes | dynamic int8 quantization of every predictor Linear + lm heads on CPU (fp32 activations), default on | micro-benchmark 70 → 36 ms per frame for the loop | +| vLLM's stock sampler spends 9 ms per step in `compiled_random_sample` (a `torch.compile`'d kernel) on CPU | talker `sample()` with `prefer_model_sampler` on CPU: same math eagerly (penalties, temperature, top-k, exponential noise with per-request generators) | 9 → <0.5 ms per step | + +Engine runs (`benchmarks/tts/stream_latency_bench.py --step-stats`, 0.6B CustomVoice, 12 prompts, NUMA node 1 pinned; per-frame numbers are the stage-0 engine step and its parts): + +| variant | threads | talker step ms/frame | talker fwd | predictor loop (15 sub-steps) | sampler | TTFA p50 | playback-start p50 | RTF | +|---|---|---|---|---|---|---|---|---| +| before (re-prefill predictor, stock sampler) | 12+4 | 134 | 25 | 97 (6.4/sub-step) | 10 | 273 ms | 4.9 s | 1.7 | +| KV-cached predictor | 12+4 | 108 | 26 | 71 (4.7) | 9 | 251 ms | 3.6 s | 1.38 | +| + predictor 4 / 2 OpenMP threads | 12+4 | 108 | 25 | 71 (4.6–4.7) | 9 | 241–254 ms | 3.5–3.6 s | 1.36–1.39 | +| + per-submodule torch.compile (rejected) | 12+4 | 131–139 | 26 | 94–101 (6.2–6.7) | 9 | 266–272 ms | 4.7 s | 1.62–1.63 | +| + compiled shape-static step (bf16) + fast sampler | 12+4 | 114 | 26 | 84 (5.6) | 0.6 | 240 ms | 3.1 s | 1.27 | +| + fast CPU sampler (bf16 KV-cached predictor) | 12+4 | 99 | 25 | 71 (4.7) | 0.6 | 245 ms | 3.6 s | 1.39 | +| + int8-dynamic predictor + fast sampler (rejected for fidelity, below) | 12+4 | 68 | 26 | 38 (2.5) | 0.7 | 212 ms | 2.0 s | 0.98 | +| int8-dynamic predictor, 24+8 threads (rejected for fidelity, below) | 32 | 59 | 21 | 34 (2.2) | 0.6 | 175 ms | 1.7 s | 0.80 | +| same, 6+2 threads | 8 | 144 | 39 | 102 (6.8) | 0.6 | 372 ms | 5.8 s | 1.92 | +| **fp32 predictor + compiled static step + fast sampler (faithful; the new default)** | 12+4 | **93** | 25 | **64 (4.3)** | 0.6 | **225 ms** | 2.7 s | **1.15** | +| same, 24+8 threads | 32 | **77** | 21 | 53 (3.5) | 0.7 | **201 ms** | 1.9 s | **0.88** | +| int8-dynamic predictor, Base checkpoint with reference speaker (x-vector) | 12+4 | 80 | 26 | 38 (2.5) | 0.7 | 696 ms (incl. per-request x-vector) | 2.6 s | 1.12 (was 1.77) | +| **fp32 default, Base checkpoint with reference speaker (x-vector), the row the llama.cpp comparison uses** | 12+4 | **109** (mean; p90 158) | 25 | 69 (4.6) | <1 | 686 ms (incl. per-request x-vector) | 3.0 s | **1.27** (was 1.77) | +| same, 24+8 threads | 32 | **91** (mean; p90 111) | 21 | 56 (3.7) | <1 | 486 ms | 2.2 s | **0.99** (was 1.7) | + +Folding the per-sub-step lm head, top-k Gumbel sampling, embedding and projection into a second compiled graph (branch commit `3085f501`) changes little on this host (sub-step 4.25 → 4.16 ms); the compiled fp32 transformer step itself is now the cost, i.e. fp32 weight traffic (4.8 GB per frame for the 15 passes), which is why the 6+2-thread class stays at 145–148 ms per frame (RTF 1.9): there the answer is a real int8/int4 GEMV kernel (PyTorch's `_weight_int8pack_mm` on this build is an unoptimized reference, 10× slower than fp32) or llama.cpp-style quantized weights, not more compilation. + +Against llama.cpp on the same host and threads (native C++, Q4_0/f16 GGUF, [edge_engine_comparison.md](edge_engine_comparison.md) §4): 85–97 ms per frame for talker + predictor plus a 55 ms/frame serial vocoder, RTF 1.9–2.0 at 8–32 threads. With the faithful fp32 predictor the branch generates a frame in 77–93 ms including sampling, with the vocoder in a concurrent stage: **RTF 0.88 (real time) on 32 threads and 1.15 on 16 threads, against llama.cpp's 1.9–2.0 at any thread count**, i.e. 1.7–2.3× faster end-to-end, and at parity per talker frame at 16 threads (93 vs 85–97 ms). The remaining terms are the talker forward (21–26 ms, vLLM's bf16 CPU backend) and the eager glue in each sub-step (now compiled; 2 % gain). + +Fidelity, measured in situ (`VLLM_OMNI_PREDICTOR_PARITY_CHECK=1`: every sub-step also runs the untouched bf16 predictor on the same inputs and compares logits): + +GPU regression check after the CPU changes (all of them are gated on the CPU platform: `_setup_cpu_fast_path` and `prefer_model_sampler` are no-ops on CUDA): 0.6B CustomVoice, edge profile, one H200 through the scheduler, 12 prompts: TTFA p50 44.5 ms, playback-start 44.5 ms, stall 0, RTF 0.086, init 77.6 s — inside the 40.7–54 ms / 0.079–0.097 band of the 2026-09-09 edge-profile runs (`results/gpu_sanity/`). + +| predictor variant vs bf16 reference | top-1 agreement | top-5 overlap | KL(ref‖test) | max \|logit diff\| | +|---|---|---|---|---| +| fp32 weights, compiled static step (control: only bf16 rounding differs) | 93.8 % | 95.3 % | 0.0015 | 0.23 | +| torch.ao dynamic int8 (per-tensor int8 activations, per-channel int8 weights) | **22.2 %** | – | – | 7.0 | +| int8 transformer body, fp32 lm heads (`ao-nohead`) | 23.7 % | 33.3 % | 0.76 | 7.0 | + +So the fast int8 rows above are **not usable as-is**: per-tensor activation quantization of the residual stream destroys the residual-codebook predictions even though the same quantization is exact on a random model (the real activations have outliers). torchao's dynamic-activation path is faithful in eager mode but numerically wrong under `torch.compile` on CPU (cosine 0.02 vs fp32, it caused 160 s runaway utterances), and its weight-only path is faithful but slow (78 ms per frame). The shipped default is therefore **fp32 weights + compiled static step** (`VLLM_OMNI_PREDICTOR_INT8=fp32`); int8 stays opt-in for measurement. + +## 4. Mobile: what can be claimed, and what cannot + +What can be claimed for mobile after this session, and what cannot: + +- **Claimable**: both halves of the on-device TTS path now exist as HTP-ready artifacts with verified numerics — the Code2Wav decoder (fp16 DLC, HTP graph prepared for v75, bit-exact on the QNN CPU backend) and the code predictor body (fp16 DLC, executed on the HTP simulator at 50.7 dB vs fp32). The talker backbone (28 layers, a standard Qwen3 decoder) is the same class of graph as the predictor and the remaining export; the codec-token protocol (WP5) and the branch's decoder export already give the phone client its contract. No other engine in this study runs Qwen3-TTS on a Hexagon NPU today (llama.cpp and MNN target the CPU/GPU on Android), so on the NPU the branch's artifacts are ahead in *deployability*. +- **Not claimable**: device speed. The x86 HTP backend is a functional simulator; its profiler reports simulator cycles, and Qualcomm AI Hub (real devices) is a cloud service that needs an account/API token this host does not have. A "faster than MNN/llama.cpp on a phone" statement therefore cannot be measured here; the honest projection is qualitative: the predictor's 15 sequential passes are exactly the kind of small-batch, launch-bound work an NPU with a prepared context binary executes in a few milliseconds each, which is what would turn the CPU-side 63–100 ms predictor loop into a sub-frame cost. +- **Next concrete steps**: (1) export the talker decode step with a static KV cache the same way; (2) run all three graphs on a Snapdragon device or AI Hub for cycle-accurate numbers; (3) an int8 GEMV kernel for the CPU predictor at 8 threads (the only CPU class where llama.cpp's Q4_0 talker is still faster per frame). + +## 5. Real Snapdragon phones through Qualcomm AI Hub (2026-09-10, afternoon) + +With the user's AI Hub token the same graphs were compiled to QNN context binaries and profiled on physical devices in Qualcomm's farm: **Samsung Galaxy S24 (Snapdragon 8 Gen 3)** and **Samsung Galaxy S25 (Snapdragon 8 Elite for Galaxy)**, every operator placed on the Hexagon NPU (`--compute_unit npu`; AI Hub confirms 100 % NPU placement for all builds). Times are AI Hub's `estimated_inference_time` (median of on-device runs). Scripts, job ids and the raw profile JSON (per-op cycles) are in `analysis/experiments/edge_engine_compare/qnn/aihub/`. Int-weight builds use AI Hub's compile-time quantization with random calibration unless marked *calibrated*; they are timing rows, the calibrated rows carry the fidelity. + +### 5.1 Per-component device timings + +| graph (real weights) | build | Galaxy S24 | Galaxy S25 | note | +|---|---|---|---|---| +| code predictor body, 17 tokens (§2 graph) | fp16 | 2.73 ms | 2.18 ms | 300/300 ops on NPU; the x86 simulator needed ~3 min for the same pass | +| code predictor cached step, 1 token, 16-slot cache | fp16 | 2.59 ms | 2.26 ms | same as the 17-token body: weight-streaming bound (160 MB fp16 per step) | +| same | w8a16 | 1.32 ms | 1.16 ms | ×15 sub-steps = **20 / 17 ms per frame** | +| same | w4a16 | 1.27 ms | 1.11 ms | no gain over int8: half the step is fixed overhead | +| talker decode step, 28 layers, 256-token cache, v1 export (stacked cache + select, expanded GQA) | fp16 | 39.7 ms | 36.9 ms | only 22–24 % of cycles in the linear layers | +| same v1 | w8a16 / w4a16 | 30.0 / 30.2 ms | 23.6 / 23.6 ms | int4 = int8: overhead-bound | +| **talker decode step, v2 export** (per-layer cache I/O, GQA by reshaping q, no cache select) | fp16 | **17.9 ms** | **15.8 ms** | 72 % of cycles in the linear layers, i.e. bandwidth-bound | +| same v2 | **w8a16** | **12.1 ms** | **9.35 ms** | | +| Code2Wav decoder chunk, 25 frames (2.0 s) with 72-frame context | fp16 | 5,197 ms | 4,539 ms | **208 / 182 ms per emitted frame**: 76 % of cycles in the Snake activation (mul·sin·pow) on 96–192 ch × 62k–186k samples | +| same | w8a16 | 4,350 ms | 4,876 ms | 16-bit activations keep the slow elementwise path | +| same | int8 (w8a8, random calibration) | **301 ms** | **266 ms** | 12.0 / 10.6 ms per emitted frame; 17× faster than fp16 | +| same, 24-frame context (49-frame window) | int8 (random calibration) | 140 ms | 118 ms | 2.2× faster, but **not a valid optimization**: truncating the context to 24 frames costs 15.9 dB against the 72-frame output on the host (48 frames: 23.0 dB, 36: 19.6, 16: 12.2), so the window is doing real work. The fp16 build of this export also failed AI Hub's context-binary conversion (exit code 15). | + +Memory: every build peaks at 100–250 MB of working memory on top of the memory-mapped context binary (talker fp16 context ≈ 0.9 GB, int8 ≈ 0.45 GB; decoder fp16 ≈ 0.2 GB). + +### 5.1b The NPU is the wrong unit for the vocoder (Adreno GPU, same phone) + +The vocoder profile said the cost was elementwise math on sample-rate tensors, which is a bandwidth problem, not a matrix problem — so the same graphs were compiled to TFLite and profiled on the **Adreno GPU** and the CPU of the Galaxy S25: + +| graph, fp16/fp32 TFLite | Hexagon NPU (QNN) | Adreno GPU | phone CPU | +|---|---|---|---| +| Code2Wav decoder chunk (2.0 s) | 4,539 ms | **542 ms** | 2,735 ms | +| → per emitted 80 ms frame | 182 ms | **21.7 ms** | 109 ms | +| talker decode step (v2) | **15.8 ms** | 29.9 ms | 29.8 ms | +| code predictor cached step | **2.25 ms** | 4.33 ms | 4.82 ms | + +The GPU runs the vocoder **8.4× faster than the NPU**, and the NPU runs the transformer steps ~2× faster than the GPU. The right deployment is therefore heterogeneous, and it maps onto the branch's existing two-stage pipeline exactly: **stage 0 (talker + code predictor) on the NPU, stage 1 (Code2Wav) on the GPU**, which on this phone already run in separate processes on the server. + +Fidelity and precision on the GPU. The TFLite GPU delegate runs fp16 by default; `--tflite_options` controls that, and the choice is a real one (Galaxy S25, same graph, same input): + +| GPU configuration | per 2 s chunk | per emitted frame | SNR vs torch fp32 | +|---|---|---|---| +| default (fp16 tensors) | 542–572 ms | 21.7–22.9 ms | 27.5 dB (corr 0.9991) | +| `allow_fp32_as_fp16=false` | 1,145 ms | **45.8 ms** | **108.4 dB** (corr 0.99999999999) | +| `gpu_inference_priority1=…MAX_PRECISION` | 2,882 ms | 115.3 ms | 106.6 dB — slower, no more accurate | +| `qnn_gpu_precision=kGpuFp32` / `kGpuFp16` | 569–580 ms | 22.8–23.2 ms | 27.5 dB — the flag changed nothing here | +| ONNX Runtime on the GPU (`--target_runtime onnx`) | 3,063 ms | 122.5 ms | 115.6 dB | +| (reference) TFLite on the phone CPU | 2,735 ms | 109 ms | 106.6 dB | + +So the vocoder can run **numerically exact on the Adreno GPU at 45.8 ms per frame** — still 4× faster than fp16 on the NPU — and there is no reason to accept the fp16 delegate's −27 dB noise floor unless the extra 24 ms per frame matters. + +### 5.1c Which mobile *runtime*, on the same graphs and the same phone + +The other half of "compared with other engines": llama.cpp cannot reach a phone's accelerators for this model at all (no Hexagon backend, CPU-only vocoder), so on-device the meaningful comparison is between the runtimes that can execute these graphs. All rows are the same two exported graphs on the Galaxy S25, fp16 unless noted, per 80 ms frame: + +| runtime | talker step, NPU | talker step, GPU | talker step, CPU | vocoder, NPU | vocoder, GPU | vocoder, CPU | +|---|---|---|---|---|---|---| +| QNN context binary | **15.8 ms** | – | – | 182 ms | – | – | +| TensorFlow Lite | 16.3 ms | 29.9 ms | 29.8 ms | fails to run after compiling | **21.7 ms** (fp16) / **45.8 ms** (fp32, exact) | 109 ms | +| ONNX Runtime | 16.6 ms | – | 33.1 ms | out of device memory | 122.5 ms | 124.3 ms | + +Reading: on the NPU all three runtimes land within 5 % of each other, because they all end up in QNN — the runtime choice does not matter there, the *export layout* does (§5.1). The runtimes separate on the other units, and the one decision worth making is the vocoder's: TensorFlow Lite on the GPU is 2.7× faster than ONNX Runtime on the same GPU and 8.4× faster than QNN on the NPU. Neither TFLite nor ONNX Runtime can run the vocoder on the NPU at all (one fails after compiling, the other exceeds device memory), so the QNN context binary is the only NPU vocoder path — and it is the slowest option of all. + +### 5.1d Beyond the two phones: laptop, automotive and IoT Snapdragons + +The same two graphs on other Snapdragon classes in the AI Hub farm (talker step as a QNN context binary on the NPU, +vocoder as TFLite on the GPU): + +| device (class) | talker step, NPU | vocoder, GPU (per emitted frame) | +|---|---|---| +| Galaxy S25 — 8 Elite (phone) | 15.8 ms | 21.7 ms | +| Galaxy S24 — 8 Gen 3 (phone) | 17.9 ms | – | +| Snapdragon X Elite CRD (Windows laptop) | 19.6 ms | TFLite is not a supported target for this device | +| SA8775P ADP (automotive) | 23.8 ms | 68.0 ms | +| Dragonwing RB3 Gen 2 — QCS6490 (low-end IoT) | rejected: "Tensor 'x' has a floating-point type which is not supported by the targeted device" | 124.3 ms | + +Reading: the transformer step is within 1.5× across a phone, a laptop and an automotive board, so the NPU story +generalises; the QCS6490 tier is integer-only, which is exactly the case where the (currently broken) int8 path would +be mandatory rather than optional. The vocoder's GPU cost scales with the GPU tier, and only the two phone GPUs are +fast enough to keep it under the 80 ms frame budget. + +### 5.2 Per-frame budget on the phone (measured components, 80 ms frame) + +| configuration (Galaxy S25) | talker | predictor ×15 | vocoder | serial | RTF | pipelined | RTF | +|---|---|---|---|---|---|---|---| +| everything on the NPU, fp16 | 15.8 | 33.9 | 182 | 232 ms | 2.9 | 182 ms | 2.3 | +| NPU steps + **GPU vocoder, fp32 (exact)** | 15.8 | 33.9 | **45.8** | 96 ms | 1.19 | **49.7 ms** | **0.62** | +| NPU steps + GPU vocoder, fp16 delegate | 15.8 | 33.9 | 21.7 | 71 ms | 0.89 | **49.7 ms** | **0.62** | +| everything on the GPU (fp16) | 29.9 | 65.0 | 21.7 | 117 ms | 1.5 | 95 ms | 1.2 | + +The pipelined column is what the branch's architecture gives: the talker stage and the Code2Wav stage are separate engines exchanging codec tokens, so on a phone they overlap and the frame cost is the slower stage, not the sum. In that mode the talker stage is the bottleneck at 49.7 ms, the vocoder is free either way, and **the exact-precision vocoder costs nothing** — so the deliverable configuration is talker + predictor on the NPU in fp16 and Code2Wav on the GPU in fp32, at **RTF 0.62 on a Snapdragon 8 Elite, real time with a 1.6× margin, with no quantization anywhere**. + +Earlier all-int8 numbers (37–44 ms per frame) are not deliverable: every automatic integer build here is numerically broken (§5.3). They remain as an upper bound on what a proper quantization effort could add. + +Not in the budget: host-side embedding lookups, sampling and the RVQ codebook sum (sub-millisecond on a phone CPU), and per-call dispatch overhead (0.1–0.3 ms × 17 calls per frame on QNN). Each graph was measured in isolation, so these are sums or maxima of medians, not a pipeline measurement. + +### 5.3 Fidelity on device + +On-device inference jobs with the same inputs as the x86 runs (`decoder_quant_chain.py`, outputs in `aihub/out_*.npy`): + +| graph, build | device | SNR vs torch fp32 | corr | +|---|---|---|---| +| Code2Wav decoder chunk, fp16 | Galaxy S25 / S24 | **39.2 / 39.1 dB** | 0.99994 | +| code predictor body, fp16 | Galaxy S25 | **48.3 dB** | 0.999993 (x86 simulator: 50.7 dB) | +| code predictor body, w8a16 / w4a16, AI Hub compile-time quantization (random calibration) | Galaxy S25 | 1.0 / 0.3 dB | 0.61 / 0.54: the random-data activation ranges are wrong, so these timing builds are not fidelity evidence | +| code predictor body, w8a16 / w8a8, quantize job calibrated on 6 draws of the input distribution (held-out test input) | Galaxy S25 | **3.1 / 0.7 dB** | 0.77 / 0.39; profiles at 2.09 / 1.23 ms | +| Code2Wav decoder, int8 (w8a8) calibrated on the test window (AI Hub quantize job, min–max ranges) | S25 / S24 | **−3.3 dB** | 0.00 (real-looking signal, uncorrelated with the reference) | +| same, calibrated on 7 held-out windows of real codec tokens (696 frames from the WP5 codec-stream runs) | S25 / S24 | **−4.3 / −4.3 dB** | 0.00 (same failure; the calibrated build profiles at 282 ms on the S25) | + +So **fp16 on the Hexagon NPU is faithful and every integer build produced here is not**: + +- fp16: decoder 39 dB (the fp16 accumulation limit of a 481-GFLOP vocoder; the same DLC on the QNN CPU backend was 107 dB), predictor 48 dB. Both are usable. +- The vocoder cannot take 8-bit activations at all: per-tensor ranges cannot hold the Snake activation (`x + sin²(ax)/a`) on sample-rate tensors, and calibrating on real codec-token windows does not help (−4.3 dB, output uncorrelated with the reference). The w8a16 build keeps the fidelity story open but runs as slowly as fp16 (4.4–4.9 s per chunk), so it buys nothing. +- The code predictor is damaged by **int8 weights alone**: calibrated w8a16 (int8 weights, int16 activations, correct activation ranges) reaches only 3.1 dB / corr 0.77. This is the same failure the CPU experiment found for torch.ao per-tensor int8 (22 % top-1 agreement, §3) and it is a property of the model, not of the runtime: the predictor's residual heads need per-channel scales or quantization-aware training. + +The x86 simulator run of the fp16 decoder (§2) was stopped after 15 h once this device parity existed. + +### 5.4 Reading + +- **The export mattered more than the device.** The first talker export spent 70 % of its NPU cycles on KV-cache plumbing; the v2 layout (per-layer cache I/O, grouped-query attention by reshaping the query) is 2.2× faster and bandwidth-bound. That layout is what the branch's HTP export should emit. +- **The unit mattered even more.** The Hexagon NPU is 8.4× *slower* than the Adreno GPU on the vocoder (whose cost is elementwise Snake math on sample-rate tensors) and ~2× *faster* on the transformer steps. Sending everything to the NPU — the obvious reading of "run it on the phone's AI accelerator" — is the worst of the three placements measured. +- **What runs faithfully today**: talker + predictor on the NPU (fp16, 39–48 dB) and Code2Wav on the GPU in **fp32** (108 dB, i.e. exact), ≈96 ms per 80 ms frame serially and **≈50 ms pipelined** on a Snapdragon 8 Elite. That is real time with a 1.6× margin, with no quantization and no precision compromise. The export code is in `vllm_omni/edge/qnn_export.py` (transformer steps) and `vllm_omni/edge/decoder_export.py` (vocoder). +- **What still needs work**: a stateful streaming vocoder would cut the GPU's 21.7 ms further (the chunk synthesizes 3.9× more samples than it emits, and simply shortening the context is not valid — 24 frames of context costs 15.9 dB); int8 needs per-channel scales or QAT to be usable at all. +- **Against the other engines on mobile**: llama.cpp and MNN have no Hexagon NPU path for Qwen3-TTS, and llama.cpp's CPU-only design extrapolates above RTF 2 on a phone's CPU cores (1.9–2.0 on a 16–32-thread Xeon). The branch is the only engine here with a validated on-device accelerator path, and with the NPU/GPU split it is real time on the phone while llama.cpp is not. diff --git a/docs/engines/ncnn.md b/docs/engines/ncnn.md new file mode 100644 index 000000000..1ea5b7db6 --- /dev/null +++ b/docs/engines/ncnn.md @@ -0,0 +1,524 @@ +# ncnn (Tencent/ncnn) — Engine Analysis + +| Field | Value | +|---|---| +| Local path | `/data/zhoutaichang/embedding_infer/ncnn` | +| Remote | `https://github.com/Tencent/ncnn.git` (origin) | +| Branch | `master` | +| HEAD | `3b7bdba7fc8aea8fd46779533eee027df77c639d` — "Fix SIGFPE with missing or incomplete CPU topology (#6967)", 2026-09-07 | +| Dirty status | clean (`git status --porcelain` empty) | +| Submodules | **not initialized**: `glslang` (780b6a20…) and `python/pybind11` (d03662f0…) show `-` in `git submodule status` | +| Latest tag in checkout | `20260526` (README links the 20260526 release archives) | +| Analysis date | 2026-09-08 | +| Repo AGENTS.md / CLAUDE.md | none present (checked `AGENTS.md`, `CLAUDE.md`, `.github/copilot-instructions.md`) | + +Evidence labels used below: **[Verified]** read in code/docs, **[Inferred]** reasoned from structure, **[Proposal]** design suggestion, **[Unknown]** could not determine. + +--- + +## 0. Summary + +- **What it is.** ncnn is a dependency-free C++ inference *library* (not a serving stack): a static graph runtime with a `Net` (graph + weights) and `Extractor` (per-request lazy evaluator) API, hand-optimized CPU kernels (ARM NEON/fp16/bf16/dotprod/i8mm/SVE, x86 SSE2→AVX512-FP16/VNNI/BF16, RISC-V V/Zfh, MIPS MSA, LoongArch LSX/LASX) and a Vulkan compute GPU backend. [Verified] ([README](../../ncnn/README.md), [src/net.h](../../ncnn/src/net.h), [src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt)) +- **No serving layer.** There is no HTTP server, request queue, scheduler, or continuous batching; the "runtime" is `Net::load_param/load_model` + `Extractor::input/extract`. Request-level scheduling is entirely the application's job. [Verified] (§5) +- **Graph execution is lazy pull-based.** `Extractor::extract(blob)` recursively runs `NetPrivate::forward_layer` on the producer chain until the requested blob is materialized; unrequested branches never run. [Verified] ([src/net.cpp#L123](../../ncnn/src/net.cpp#L123), [#L2871](../../ncnn/src/net.cpp#L2871)) +- **Backends in tree:** CPU (with runtime ISA dispatch) and Vulkan only. No CUDA, Metal, CoreML, OpenCL, NNAPI, QNN. Apple GPUs are reached through MoltenVK (`libMoltenVK.dylib` is dlopen'ed by the in-house loader). [Verified] ([src/simplevk.cpp#L558](../../ncnn/src/simplevk.cpp#L558), [FAQ-ncnn-vulkan.md](../../ncnn/docs/how-to-use-and-FAQ/FAQ-ncnn-vulkan.md)) +- **Transformer/LLM support is real and recent (2025–2026).** Layers `MultiHeadAttention`, `SDPA` (with GQA and flash-attention style CPU/Vulkan kernels), `RotaryEmbed`, `RMSNorm`, `Embed`, `Gemm`, `GLU`, `GELU` exist with arch-specific kernels; an in-layer **KV cache** (`7=1` param, `cache_k/v_in/out` blobs, `Extractor::set_kvcache_allocator`, `set_kvcache_max_seqlen_hint`, `extract(..., type=1)`) is documented and tested. [Verified] ([docs/developer-guide/kvcache.md](../../ncnn/docs/developer-guide/kvcache.md), [src/layer/sdpa.cpp](../../ncnn/src/layer/sdpa.cpp), [src/net.h#L193](../../ncnn/src/net.h#L193)) +- **LLM benchmark harness ships in-tree**: `benchncnn_llm` measures 256-token prefill and cached single-token decode for Qwen2.5-0.5B, Qwen3-0.6B, MiniCPM4-0.5B, Hunyuan-0.5B, TinyLlama-1.1B, Llama3.2-1B, Youtu-LLM-2B decoders (param files only; weights are random). [Verified] ([benchmark/benchncnn_llm.cpp](../../ncnn/benchmark/benchncnn_llm.cpp), [benchmark/CMakeLists.txt](../../ncnn/benchmark/CMakeLists.txt)) +- **LLM weight quantization exists**: block-quantized Gemm/MHA (W4/W6 weight-only, W8A8 dynamic per-block on CPU) via `ncnnllm2table`/`ncnnllm2int` (minmax/mseclip/awq/gptq), gated by `NCNN_WEIGHT_QUANT`; plus classic PTQ int8 via `ncnn2table`/`ncnn2int8`. [Verified] ([docs/how-to-use-and-FAQ/quantized-int8-inference.md](../../ncnn/docs/how-to-use-and-FAQ/quantized-int8-inference.md), [tools/quantize](../../ncnn/tools/quantize)) +- **TTS exists as an example, not a runtime**: `examples/piper.cpp` runs a VITS (Piper) pipeline split into 5 ncnn nets (enc_p, emb_g, dp, flow, dec) with custom layers registered from the app, a dictionary-based English phonemizer and WAV writer. `examples/whisper.cpp` runs Whisper ASR (fbank net → encoder → cached decoder with beam search). Audio ops `Spectrogram`/`InverseSpectrogram` (STFT/iSTFT) and 1D conv/deconv exist. [Verified] ([examples/piper.cpp](../../ncnn/examples/piper.cpp), [examples/whisper.cpp](../../ncnn/examples/whisper.cpp), [src/layer/spectrogram.h](../../ncnn/src/layer/spectrogram.h)) +- **No tokenizer, no sampler, no vocoder library.** Whisper example carries a minimal reverse-vocab tokenizer; Piper carries a dict phonemizer. Nothing generic. [Verified] +- **VLA: nothing specific.** Vision encoders (CNN/ViT) are first-class; MLP/Gemm heads trivial; diffusion/flow policies would require the host to loop denoise steps calling one Extractor per step (no scheduler code). [Verified]/[Inferred] (§10) +- **Model format**: text `.param` + flat `.bin` (magic 7767517), produced mainly by **pnnx** (PyTorch/TorchScript/ONNX → ncnn; pip installable) with fp16 weights by default; legacy converters for Caffe/ONNX/Darknet/MXNet/Keras/TF-MLIR. [Verified] ([docs/developer-guide/param-and-model-file-structure.md](../../ncnn/docs/developer-guide/param-and-model-file-structure.md), [tools/pnnx/README.md](../../ncnn/tools/pnnx/README.md)) +- **Dynamic shapes** are natural: every `Mat` carries its own dims; layers compute output shapes at forward time; param shape hints (`30=`) are optional; `Reshape` supports runtime **expressions** over input shapes. [Verified] ([docs/developer-guide/expression.md](../../ncnn/docs/developer-guide/expression.md)) +- **Threading** is OpenMP-inside-kernels (`#pragma omp parallel for num_threads(opt.num_threads)`), with big.LITTLE affinity via `set_cpu_powersave`, a bundled minimal OpenMP runtime (`simpleomp`) and a bundled Vulkan loader (`simplevk`) so no external runtime deps are needed. [Verified] ([src/cpu.h](../../ncnn/src/cpu.h), [src/simpleomp.h](../../ncnn/src/simpleomp.h), [docs/developer-guide/vulkan-driver-loader.md](../../ncnn/docs/developer-guide/vulkan-driver-loader.md)) +- **Vulkan shaders are GLSL embedded as source and compiled to SPIR-V at runtime** by the in-tree glslang, with a persistent `PipelineCache` (SPIR-V + driver blob) to cut startup. [Verified] ([cmake/ncnn_add_shader.cmake](../../ncnn/cmake/ncnn_add_shader.cmake), [docs/developer-guide/vulkan-pipeline-cache.md](../../ncnn/docs/developer-guide/vulkan-pipeline-cache.md)) +- **Edge portability is the strongest asset**: 60 CMake toolchains (Android, iOS/tvOS/visionOS/watchOS via ios.toolchain, HarmonyOS, Jetson, QNX, ESP32, RISC-V bare-metal, HiSilicon, Loongson, MIPS, PowerPC, Windows XP, WebAssembly), 45 CI workflows, `NCNN_SIMPLESTL/SIMPLEOMP/SIMPLEOCV/SIMPLEMATH` shims, per-layer `WITH_LAYER_x` opt-out for size. [Verified] ([toolchains](../../ncnn/toolchains), [docs/how-to-use-and-FAQ/build-minimal-library.md](../../ncnn/docs/how-to-use-and-FAQ/build-minimal-library.md)) +- **Batching**: `NCNN_BATCH` adds an `n` dimension to `Mat`; layers lacking `support_batch` are looped per item by the Net. KV cache supports **batch size 1 only**. [Verified] ([src/net.cpp#L662](../../ncnn/src/net.cpp#L662), [#L2882](../../ncnn/src/net.cpp#L2882)) + +--- + +## 1. Architecture overview + +```mermaid +graph TD + subgraph App["Application (C++/C/Python/JNI)"] + A1[Mat::from_pixels / audio PCM / token ids] --> A2[Net.load_param + load_model] + A2 --> A3[Extractor.input / extract] + end + + subgraph Runtime["ncnn core (src/)"] + N[Net / NetPrivate
layers[], blobs[], custom registry] --> E[Extractor / ExtractorPrivate
blob_mats[], blob_mats_gpu[], Option] + E -->|lazy pull| FL[NetPrivate::forward_layer] + FL --> CL[convert_layout
fp16/bf16 cast + elempack] + CL --> DL[do_forward_layer
batch loop, lightmode inplace] + DL --> L[Layer_final
layer_cpu / layer_vulkan] + O[Option
num_threads, use_fp16_*, use_int8_*, use_bf16_storage, use_packing_layout, allocators, kvcache_*] --> FL + P[ParamDict / ModelBin / DataReader] --> N + AL[Allocator / PoolAllocator / UnlockedPoolAllocator] --> E + end + + subgraph CPU["CPU backend"] + L --> LC[layer/*.cpp naive] + L --> LA[layer/arm|x86|riscv|mips|loongarch
runtime ISA dispatch] + CPUI[cpu.cpp: cpu_support_*, big.LITTLE, affinity, omp] --> LA + end + + subgraph GPU["Vulkan backend (NCNN_VULKAN)"] + L --> LV[layer/vulkan/*_vulkan.cpp] + LV --> PL[Pipeline / PipelineCache
glslang GLSL->SPIR-V at runtime] + LV --> CMD[VkCompute / VkTransfer
record_upload/download/pipeline, submit_and_wait] + CMD --> VD[VulkanDevice / GpuInfo
queues, VkAllocator pools] + VD --> SVK[simplevk loader
dlopen libvulkan / MoltenVK / ICD] + end + + subgraph Tools["Offline tools"] + T1[pnnx: PyTorch/ONNX -> .ncnn.param/.bin] --> A2 + T2[ncnnoptimize: fuse/eliminate, fp16 weights] --> A2 + T3[ncnn2table/ncnn2int8, ncnnllm2table/ncnnllm2int] --> A2 + end +``` + +**Key modules** [Verified] + +| Module | Path | Responsibility | Key symbols | +|---|---|---|---| +| Graph container | [src/net.h](../../ncnn/src/net.h), [src/net.cpp](../../ncnn/src/net.cpp) | Parse param, create layers (overwrite → vulkan → cpu → custom), load weights, create pipelines, upload to GPU, own local pool allocators | `Net`, `NetPrivate`, `Net::load_param` (L1306), `Net::load_model` (L2022), `Net::register_custom_layer` (L1215), `create_extractor` | +| Per-request executor | [src/net.cpp#L2676](../../ncnn/src/net.cpp#L2676) | Holds `blob_mats` (+GPU), per-extractor `Option`; lazy DAG evaluation; unpack/cast outputs | `Extractor`, `ExtractorPrivate`, `Extractor::extract` (L2871), `NetPrivate::forward_layer` (L123 CPU, L192 Vulkan), `convert_layout` (L358), `do_forward_layer` (L633) | +| Layer base + registry | [src/layer.h](../../ncnn/src/layer.h), [src/layer.cpp](../../ncnn/src/layer.cpp) | Virtual op interface, capability flags, factory tables | `Layer`, `support_*` flags (L45–90), `forward/forward_inplace`, `Layer_final` (L201), `layer_registry`, `create_layer_cpu/naive/vulkan`, `DEFINE_LAYER_CREATOR` (L199) | +| Tensor | [src/mat.h](../../ncnn/src/mat.h) | N-D tensor with `elemsize/elempack/cstep/n`; pixel conversion | `Mat` (L50), `VkMat` (L389), `VkImageMat` (L559), `from_pixels*` (L293–307), `substract_mean_normalize` (L335), `batch()` (L239) | +| Options | [src/option.h](../../ncnn/src/option.h), [src/option.cpp](../../ncnn/src/option.cpp) | All runtime knobs incl. precision, allocators, kv cache | `Option` | +| Allocators | [src/allocator.h](../../ncnn/src/allocator.h), [src/allocator.cpp](../../ncnn/src/allocator.cpp) | Aligned malloc, host pools, Vulkan buffer/image pools | `fastMalloc` (L56), `Allocator` (L142), `PoolAllocator` (L151), `UnlockedPoolAllocator` (L180), `VkAllocator` (L267), `VkBlobAllocator`, `VkWeightAllocator`, `VkStagingAllocator`, `VkAndroidHardwareBufferImageAllocator` (L405) | +| CPU topology | [src/cpu.h](../../ncnn/src/cpu.h), [src/cpu.cpp](../../ncnn/src/cpu.cpp) | ISA feature detection (ruapu), core counts, big.LITTLE, affinity, OpenMP wrappers, denormals | `cpu_support_arm_*`, `cpu_support_x86_*`, `get_big_cpu_count`, `set_cpu_powersave`, `set_cpu_thread_affinity`, `set_kmp_blocktime`, `set_flush_denormals` | +| Vulkan device | [src/gpu.h](../../ncnn/src/gpu.h), [src/gpu.cpp](../../ncnn/src/gpu.cpp) | Instance/device creation, capability query, queue + allocator pools, shader compile | `create_gpu_instance` (gpu.cpp L2919), `GpuInfo`, `VulkanDevice`, `get_gpu_device`, `compile_spirv_module` | +| Vulkan commands | [src/command.h](../../ncnn/src/command.h), [src/command.cpp](../../ncnn/src/command.cpp) | Command buffer recording/submission, staging copies | `VkCompute::record_upload/record_download/record_pipeline/submit_and_wait`, `VkTransfer` | +| Vulkan pipelines | [src/pipeline.h](../../ncnn/src/pipeline.h), [src/pipelinecache.h](../../ncnn/src/pipelinecache.h) | Shader module/pipeline objects, on-disk cache | `Pipeline`, `PipelineCache::save_cache/load_cache/get_pipeline` | +| Vulkan loader | [src/simplevk.h](../../ncnn/src/simplevk.h), [src/simplevk.cpp](../../ncnn/src/simplevk.cpp) | In-house loader (no libvulkan link) | `load_vulkan_driver` | +| Param/weights I/O | [src/paramdict.h](../../ncnn/src/paramdict.h), [src/modelbin.h](../../ncnn/src/modelbin.h), [src/datareader.h](../../ncnn/src/datareader.h) | Key=value params, weight blobs with type flags, stream abstraction | `ParamDict`, `ModelBin`, `DataReader` | +| Operator impls | [src/layer](../../ncnn/src/layer) (110 registered types), [src/layer/arm](../../ncnn/src/layer/arm), [src/layer/x86](../../ncnn/src/layer/x86), [src/layer/vulkan](../../ncnn/src/layer/vulkan), [src/layer/riscv](../../ncnn/src/layer/riscv), [src/layer/mips](../../ncnn/src/layer/mips), [src/layer/loongarch](../../ncnn/src/layer/loongarch) | Naive reference + arch kernels + GLSL shaders | e.g. `SDPA`, `MultiHeadAttention`, `Gemm`, `Convolution` | +| Shims | [src/simpleomp.h](../../ncnn/src/simpleomp.h), [src/simplestl.h](../../ncnn/src/simplestl.h), [src/simpleocv.h](../../ncnn/src/simpleocv.h), [src/simplemath.h](../../ncnn/src/simplemath.h) | Minimal OpenMP runtime, STL, OpenCV-like image I/O, libm | — | +| C API | [src/c_api.h](../../ncnn/src/c_api.h) | Flat C bindings incl. kv cache options | `ncnn_net_*`, `ncnn_extractor_*`, `ncnn_option_set_kvcache_allocator` | +| Python | [python/src/main.cpp](../../ncnn/python/src/main.cpp) | pybind11 bindings for Net/Extractor/Mat/Option/Layer/Vulkan | `py::class_` (L959) | +| Converters | [tools/pnnx](../../ncnn/tools/pnnx), [tools/onnx](../../ncnn/tools/onnx), [tools/ncnnoptimize.cpp](../../ncnn/tools/ncnnoptimize.cpp), [tools/quantize](../../ncnn/tools/quantize) | Model export, offline fusion, quantization | `pnnx`, `onnx2ncnn`, `ncnnoptimize`, `ncnn2table`, `ncnn2int8`, `ncnnllm2table`, `ncnnllm2int` | +| Build glue | [cmake/ncnn_add_layer.cmake](../../ncnn/cmake/ncnn_add_layer.cmake), [cmake/ncnn_add_shader.cmake](../../ncnn/cmake/ncnn_add_shader.cmake), [src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt) | Layer registry generation, ISA-variant source generation, shader embedding | `ncnn_add_layer`, `ncnn_add_arch_opt_layer`, `ncnn_add_shader` | + +--- + +## 2. Device layer + +### Backends actually in tree [Verified] +- **CPU**: naive C++ in `src/layer/*.cpp` plus arch dirs: arm (56 `*_arm.cpp` + ISA variants `_asimdhp/_asimddp/_asimdfhm/_bf16/_i8mm/_vfpv4`), x86 (59 base + `_avx2/_fma/_fma4/_xop/_f16c/_avx512*/_avxvnni*/_avxneconvert`), riscv (52), mips (56), loongarch (56). Counts from `ls src/layer//*_.cpp`. +- **Vulkan GPU**: 64 `*_vulkan.cpp` layer implementations and 300 `.comp` shaders under [src/layer/vulkan/shader](../../ncnn/src/layer/vulkan/shader). Enabled by `NCNN_VULKAN` (default **OFF**) ([CMakeLists.txt#L82](../../ncnn/CMakeLists.txt#L82)). +- **Not present**: CUDA, Metal, CoreML, OpenCL, NNAPI, QNN/Hexagon, TensorRT. A grep for those names over `src/` only hits Vulkan loader/portability code (`VK_KHR_portability_subset`, MoltenVK dlopen). The FAQ explicitly states Vulkan was chosen over CUDA/OpenCL/Metal for portability ([FAQ-ncnn-vulkan.md](../../ncnn/docs/how-to-use-and-FAQ/FAQ-ncnn-vulkan.md), "why using vulkan over cuda/opencl/metal"). + +### Device discovery / selection [Verified] +- CPU: `get_cpu_count / get_big_cpu_count / get_little_cpu_count / get_physical_*` ([src/cpu.h#L124](../../ncnn/src/cpu.h#L124)); big.LITTLE is derived from sysfs `cpufreq` max frequency (`get_max_freq_khz`, [src/cpu.cpp#L1486](../../ncnn/src/cpu.cpp#L1486)) and topology files (`thread_siblings`, [src/cpu.cpp#L961](../../ncnn/src/cpu.cpp#L961)). ISA features: `cpu_support_arm_asimdhp/asimddp/bf16/i8mm/sve*`, `cpu_support_x86_avx2/avx512_*`, `cpu_support_riscv_v/zfh/zvfh`, `cpu_riscv_vlenb` via bundled [src/ruapu.h](../../ncnn/src/ruapu.h). `Option::num_threads` defaults to `get_physical_big_cpu_count()` ([src/option.cpp](../../ncnn/src/option.cpp)). +- Vulkan: lazy singleton `create_gpu_instance()` under a mutex ([src/gpu.cpp#L2919](../../ncnn/src/gpu.cpp#L2919)); loader order: env `VK_ICD_FILENAMES`/`NCNN_VULKAN_DRIVER` → system `libvulkan.so/.dylib/vulkan-1.dll` → direct driver file search (`libvulkan_radeon.so`, `libMoltenVK.dylib`, Android HAL `HMI`) ([src/simplevk.cpp#L287](../../ncnn/src/simplevk.cpp#L287), [#L558](../../ncnn/src/simplevk.cpp#L558), [vulkan-driver-loader.md](../../ncnn/docs/developer-guide/vulkan-driver-loader.md)). Each physical device gets a `GpuInfo` with a `rough_score` (device type, subgroup size, fp16/int8/coopmat support, memory) used to choose `get_default_gpu_index()` ([src/gpu.cpp#L1297](../../ncnn/src/gpu.cpp#L1297), [#L3624](../../ncnn/src/gpu.cpp#L3624)). `Net::set_vulkan_device(int|VulkanDevice*)` or `opt.vulkan_device_index` selects a device before `load_param` ([src/net.cpp#L1338](../../ncnn/src/net.cpp#L1338)). If no valid device, `opt.use_vulkan_compute` is silently set to false (CPU fallback) ([src/net.cpp#L1348](../../ncnn/src/net.cpp#L1348)). +- Capability sanitization: at `load_param`, unsupported `use_fp16_packed/storage/arithmetic`, `use_int8_*`, `use_bf16_*`, `use_cooperative_matrix`, `use_subgroup_ops` are cleared from the Net option based on `GpuInfo` ([src/net.cpp#L1350–1372](../../ncnn/src/net.cpp#L1350)). `GpuInfo` exposes ~50 extension flags including `VK_KHR_cooperative_matrix`, `VK_NV_cooperative_matrix2`, `VK_KHR_shader_bfloat16`, `VK_EXT_shader_float8`, `VK_ANDROID_external_memory_android_hardware_buffer` ([src/gpu.h#L327–378](../../ncnn/src/gpu.h#L327)). + +### Memory allocation / pools [Verified] +- Host: `fastMalloc` = aligned alloc with `NCNN_MALLOC_ALIGN` and 64-byte over-read pad ([src/allocator.h#L56](../../ncnn/src/allocator.h#L56)). `PoolAllocator` (mutex) / `UnlockedPoolAllocator` keep `budgets`/`payouts` lists; reuse if a free block is ≥ size and within `size_compare_ratio`; drop when > `size_drop_threshold` (default 10) ([src/allocator.cpp#L98](../../ncnn/src/allocator.cpp#L98)). Option roles: `blob_allocator` (named blobs, lifetime = Extractor), `workspace_allocator` (per-layer scratch), `kvcache_allocator` (must differ from blob allocator, [src/net.cpp#L2876](../../ncnn/src/net.cpp#L2876)). With `use_local_pool_allocator` (default true) the Net lazily creates its own pools at `load_model` ([src/net.cpp#L2136](../../ncnn/src/net.cpp#L2136)) and clones outputs out of them on `extract` so the Net can be destroyed early ([src/net.cpp#L3060](../../ncnn/src/net.cpp#L3060)). +- Device: `VkAllocator` family: `VkBlobAllocator` (activations), `VkWeightAllocator` (device-local weights; `use_weights_in_host_memory` flips to host-visible), `VkStagingAllocator`/`VkWeightStagingAllocator` (host-visible staging), with `flush/invalidate` for non-coherent memory ([src/allocator.h#L267–395](../../ncnn/src/allocator.h#L267)). `VulkanDevice::acquire_blob_allocator/acquire_staging_allocator` hand out pooled allocators; an Extractor acquires them on first `extract` if not set ([src/net.cpp#L2915](../../ncnn/src/net.cpp#L2915)). Memory type choice via `find_memory_index(required, preferred, preferred_not)` and `is_mappable/is_coherent/is_device_local` ([src/gpu.h#L465](../../ncnn/src/gpu.h#L465)). Android `AHardwareBuffer` zero-copy import allocator exists (API ≥ 26) ([src/allocator.h#L405](../../ncnn/src/allocator.h#L405), [use-ncnn-with-android-hardware-buffer.md](../../ncnn/docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md)). +- Model weights: `Net::load_model(const unsigned char*)` references external memory without copying; `opt.use_mapped_model_loading` mmaps the `.bin` on Linux/Android/OHOS/Windows/Apple ([src/net.cpp#L2261](../../ncnn/src/net.cpp#L2261)). + +### Host↔device transfers and synchronization [Verified] +- All GPU work is recorded into a `VkCompute` command buffer: `record_upload` (staging copy + optional fp32→fp16 cast + packing), `record_download`, `record_clone`, `record_pipeline(pipeline, bindings, push constants, dispatcher)`, then `submit_and_wait()` (fence wait) ([src/command.h#L22–87](../../ncnn/src/command.h#L22)). Weights are uploaded once at `load_model` via `VkTransfer`, committed every 256 MB ([src/net.cpp#L2115](../../ncnn/src/net.cpp#L2115)). +- Inside a Vulkan extract, the Net records the whole reachable subgraph into one command buffer; it force-submits when a CPU-only layer needs a downloaded input, or when `pending_dispatch_total()` exceeds a threshold scaled by `rough_score` (32 KB … 8 MB "for avoiding driver timeout") ([src/net.cpp#L250–283](../../ncnn/src/net.cpp#L250)). Final output is downloaded and waited on in `Extractor::extract` ([src/net.cpp#L2940](../../ncnn/src/net.cpp#L2940)). Barriers are inserted automatically by `barrier_readwrite` bookkeeping ([src/command.h#L82](../../ncnn/src/command.h#L82)). +- Queues: `VulkanDevice::acquire_queue/reclaim_queue` pool per family; `GpuInfo::compute_queue_count()` is documented as the practical max number of concurrent extractors ([vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md), "batch inference"). Zero-copy `VkMat::mapped()` on unified-memory devices and cross-net `VkMat` chaining through a shared `VkCompute` are documented there too. +- Timestamps/profiling: `NCNN_BENCHMARK` build records per-layer GPU timestamps via query pool ([src/net.cpp#L2952](../../ncnn/src/net.cpp#L2952)). + +### Portability [Verified] +- OS: Linux, Android (NDK ≥ r18b; Vulkan needs API 24 with system loader, or lower with simplevk), iOS/macOS/Catalyst/tvOS/visionOS/watchOS, Windows (incl. XP), HarmonyOS (`__OHOS__`), QNX, WebAssembly (Emscripten), bare-metal RISC-V/ESP32 — evidenced by [toolchains](../../ncnn/toolchains) and [.github/workflows](../../ncnn/.github/workflows) (android.yml, ios.yml, harmonyos.yml, web-assembly.yml, esp32.yml, elf-riscv64.yml, windows-arm.yml, linux-ppc64.yml, linux-loongarch64.yml …). +- Vulkan platform matrix (repo doc): Intel/AMD/NVIDIA on Windows/Linux, Qualcomm/ARM Mali on Android, Apple on mac/iOS via MoltenVK ([vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md)). Jetson: aarch64 CPU + Tegra Vulkan driver ([docs/how-to-build/how-to-build.md#L66](../../ncnn/docs/how-to-build/how-to-build.md#L66), [toolchains/jetson.toolchain.cmake](../../ncnn/toolchains/jetson.toolchain.cmake)). + +--- + +## 3. Kernel layer + +### Operator set & registration [Verified] +- 110 layer types registered via `ncnn_add_layer(Class)` in [src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt) (two marked OFF by default: `ArgMax`, `SPP`). Full list with params in [docs/developer-guide/operators.md](../../ncnn/docs/developer-guide/operators.md) and [operation-param-weight-table.md](../../ncnn/docs/developer-guide/operation-param-weight-table.md). Transformer/audio-relevant: `MultiHeadAttention`, `SDPA`, `RotaryEmbed`, `RMSNorm`, `LayerNorm`, `GroupNorm`, `Embed`, `Gemm`, `MatMul`, `Einsum`, `GLU`, `GELU`, `Softmax`, `Convolution1D/DepthWise1D/Deconvolution1D`, `Pooling1D`, `Spectrogram`, `InverseSpectrogram`, `LSTM/GRU/RNN`, `GridSample`, `DeformableConv2D`, `Interp`, `PixelShuffle`, `CumulativeSum`. +- Registration is **build-time code generation**: `ncnn_add_layer` (macro at [cmake/ncnn_add_layer.cmake#L82](../../ncnn/cmake/ncnn_add_layer.cmake#L82)) appends `#include` + `DEFINE_LAYER_CREATOR(Class)` lines into generated `layer_declaration.h` and a `layer_registry` table per ISA level (`layer_registry_arm82`, `_avx512`, …) into `layer_registry.h` (templates: [src/layer_declaration.h.in](../../ncnn/src/layer_declaration.h.in), [src/layer_registry.h.in](../../ncnn/src/layer_registry.h.in), [src/layer_type_enum.h.in](../../ncnn/src/layer_type_enum.h.in)). ISA variants are produced by rewriting the base arch source with `cmake/ncnn_generate__source.cmake` (renaming the class, e.g. `Convolution_arm_asimdhp`) and compiling with the ISA flags ([cmake/ncnn_add_layer.cmake#L2](../../ncnn/cmake/ncnn_add_layer.cmake#L2)). `WITH_LAYER_=OFF` removes a layer and its kernels (registry gets a null creator). +- `layer_to_index(name)` is a linear strcmp scan ([src/layer.cpp#L150](../../ncnn/src/layer.cpp#L150)); `create_layer_cpu(index)` picks the best registry table at runtime based on `cpu_support_*` when `NCNN_RUNTIME_CPU=ON` (see generated `layer_registry.h` consumers in [src/layer.cpp](../../ncnn/src/layer.cpp) after L400) [Verified for mechanism; exact selection code lives in the generated portion]. + +### Dispatch mechanism (op → backend impl) [Verified] +1. `Net::load_param` creates each layer as: overwrite-builtin registry → `create_layer_vulkan` (if `use_vulkan_compute` and device) → `create_layer_cpu` → custom registry; missing type is a hard error ([src/net.cpp#L1400–1420](../../ncnn/src/net.cpp#L1400)). +2. `create_layer_vulkan/cpu` return a `Layer_final` wrapper holding `layer_cpu` and optionally `layer_vulkan`; `load_param/create_pipeline` try the Vulkan layer first and **delete it and fall back to CPU** if `support_vulkan` becomes false after `load_param` (layer-specific param not supported on GPU) ([src/layer.cpp#L280–344](../../ncnn/src/layer.cpp#L280)). The Net additionally recreates a CPU layer when the Vulkan variant rejects the params or the layer's `featmask` disables Vulkan ([src/net.cpp#L1536](../../ncnn/src/net.cpp#L1536)). +3. At forward time, `Layer_final::forward(Mat…)` → `layer_cpu`, `forward(VkMat…)` → `layer_vulkan` ([src/layer.cpp#L359–400](../../ncnn/src/layer.cpp#L359)). The Vulkan `forward_layer` inspects `layer->support_vulkan` per layer: GPU layers get `record_upload` of any host-only inputs; CPU-only layers get `record_download` + forced `submit_and_wait`, then run on host ([src/net.cpp#L215–248](../../ncnn/src/net.cpp#L215)). Mixed CPU/GPU graphs therefore work automatically at the cost of syncs. +4. Inside an arch kernel, further dispatch is by data type: e.g. `Clip_arm::forward_inplace` checks `elembits()==16 && opt.use_fp16_storage` → fp16 path, bf16 path, else fp32 ([layer-support-behavior.md](../../ncnn/docs/developer-guide/layer-support-behavior.md)). `SDPA_arm` declares `support_fp16_storage = cpu_support_arm_asimdhp()` and `support_bf16_storage = true` ([src/layer/arm/sdpa_arm.cpp#L14](../../ncnn/src/layer/arm/sdpa_arm.cpp#L14)). +5. Layout negotiation is done by the Net, not the kernel: `convert_layout` casts fp32→fp16/bf16 when `layer->support_fp16_storage/support_bf16_storage` and the option is on, and re-packs to the arch-optimal `elempack` (4 NEON, 8 AVX, 16 AVX-512, `vlenb/4` RVV) unless `support_any_packing` ([src/net.cpp#L358–470](../../ncnn/src/net.cpp#L358); [element-packing.md](../../ncnn/docs/developer-guide/element-packing.md)). KV-cache blobs skip layout conversion ([src/net.cpp#L364](../../ncnn/src/net.cpp#L364)). +6. Per-layer overrides: `31=` bits disable fp16 arithmetic/storage, bf16, int8, vulkan, sgemm, winograd, threading for that layer ([docs/developer-guide/layer-feat-mask.md](../../ncnn/docs/developer-guide/layer-feat-mask.md), [src/net.cpp#L100](../../ncnn/src/net.cpp#L100)). + +### Quantization formats & where dequant happens [Verified] +- **Weight storage flags** in `.bin`: `0` fp32, `0x01306B47` fp16, `0x01348B83` bf16, other = quantized int8 with table ([param-and-model-file-structure.md](../../ncnn/docs/developer-guide/param-and-model-file-structure.md)). pnnx writes fp16 weights by default (`fp16=1`, [tools/pnnx/src/save_ncnn.cpp#L240](../../ncnn/tools/pnnx/src/save_ncnn.cpp#L240)); `ModelBin` converts to the layer's requested type at load. +- **Activation precision**: fp16 storage/arithmetic on ARMv8.2 (`asimdhp`), VFPv4 fp16 storage on ARMv7, bf16 storage on any ARM/x86 (`NCNN_BF16`), fp16/bf16/int8 packed+storage+arithmetic on Vulkan, chosen by `Option` flags and clamped by device caps. Casting happens in `convert_layout` at layer boundaries and in `Extractor::extract` back to fp32 unless `type=1` ([src/net.cpp#L2985–3055](../../ncnn/src/net.cpp#L2985)). +- **PTQ int8** (`NCNN_INT8`): `ncnn2table` (KL/ACIQ, from images or `.npy`, also static weight scales for RNN/GRU/LSTM/MHA/Embed) → `ncnn2int8` rewrites weights and adds `8=`/`18=` int8 scale params; `Quantize/Dequantize/Requantize` layers exist and conv/innerproduct/gemm/mha/sdpa/embed have `forward_int8` paths; dequant to fp32 happens inside those layers or at `extract` (int8 output cast, [src/net.cpp#L3043](../../ncnn/src/net.cpp#L3043)). x86 int8 uses AVX-VNNI/AVX512-VNNI/AVX-VNNI-INT8 kernels; ARM uses dotprod/i8mm ([src/layer/arm/convolution_arm_i8mm.cpp](../../ncnn/src/layer/arm/convolution_arm_i8mm.cpp), [src/layer/x86/convolution_x86_avx512vnni.cpp](../../ncnn/src/layer/x86/convolution_x86_avx512vnni.cpp)). +- **LLM block quant** (`NCNN_WEIGHT_QUANT`, default ON): `Gemm`/`MultiHeadAttention` `quantize_term = bits*100 + input_scale*10 + block_code` (bits 4/6/8, block 32/64/128); W4/W6 are weight-only with fp32 activations, W8 is dynamic W8A8 per-block int32 accumulate, CPU only; when enabled the layer turns off packing/fp16/bf16/Vulkan (`support_vulkan=false`) ([src/layer/multiheadattention.cpp#L84–100](../../ncnn/src/layer/multiheadattention.cpp#L84), [src/layer/gemm.h](../../ncnn/src/layer/gemm.h), [quantized-int8-inference.md](../../ncnn/docs/how-to-use-and-FAQ/quantized-int8-inference.md)). Tools: [tools/quantize/ncnnllm2table.cpp](../../ncnn/tools/quantize/ncnnllm2table.cpp), [tools/quantize/ncnnllm2int.cpp](../../ncnn/tools/quantize/ncnnllm2int.cpp); test [tests/test_gemm_block_quant.cpp](../../ncnn/tests/test_gemm_block_quant.cpp). + +### Compilation / JIT / graph capture [Verified] +- CPU: no JIT; all kernels are AOT C++/intrinsics/inline asm, selected at runtime by ISA table. Winograd/sgemm/packing variants are selected in each layer's `create_pipeline` based on shape hints and `Option` (e.g. `use_winograd23/43/63_convolution`). +- Vulkan: GLSL `.comp` sources are preprocessed into hex headers at build time ([cmake/ncnn_add_shader.cmake](../../ncnn/cmake/ncnn_add_shader.cmake)) and **compiled to SPIR-V at runtime** by in-tree glslang with option-dependent macros (`compile_spirv_module`, [src/gpu.h#L572](../../ncnn/src/gpu.h#L572); glslang linked privately, [src/CMakeLists.txt#L341](../../ncnn/src/CMakeLists.txt#L341)). Specialization constants + push constants parameterize shapes; `PipelineCache` deduplicates pipelines in-process and persists SPIR-V plus driver `VkPipelineCache` blobs to a file ([docs/developer-guide/vulkan-pipeline-cache.md](../../ncnn/docs/developer-guide/vulkan-pipeline-cache.md)). No command-buffer replay/graph capture across extracts: each `extract` records a fresh `VkCompute`. +- No runtime graph optimization: fusions (conv+bn, conv+activation, etc.) are done offline by `ncnnoptimize` ([tools/ncnnoptimize.cpp#L42–77](../../ncnn/tools/ncnnoptimize.cpp#L42)) or pnnx (`optlevel`). `grep fuse src/net.cpp` finds nothing. + +### Backend coverage gaps (computed from file presence) [Verified] +Computed by checking, for each of the 110 `ncnn_add_layer(...)` entries in [src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt), whether `src/layer/vulkan/_vulkan.cpp` / `src/layer/arm/_arm.cpp` exist. + +| Backend | Layers **without** a dedicated kernel (run naive C++ on CPU, or CPU fallback for Vulkan) | +|---|---| +| Vulkan (64/110 covered) | ArgMax, Bias, BNLL, **Embed**, Exp, Input, Log, MVN, Power, Proposal, ROIPooling, SPP, Threshold, Tile, **RNN, LSTM, GRU**, Squeeze, ExpandDims, DetectionOutput, YoloDetectionOutput, Yolov3DetectionOutput, PSROIPooling, ROIAlign, StatisticsPooling, **Pooling1D, ConvolutionDepthWise1D, Deconvolution1D, DeconvolutionDepthWise1D**, Convolution3D, ConvolutionDepthWise3D, Pooling3D, Deconvolution3D, DeconvolutionDepthWise3D, **MatMul, Einsum, GLU**, DeformableConv2D, Fold, **GridSample**, CumulativeSum, CopyTo, Diag, **Spectrogram, InverseSpectrogram**, Flip | +| ARM NEON (56/110 covered) | ArgMax, BNLL, **Embed**, Exp, Input, Log, MemoryData, MVN, Power, Proposal, Reduction, ROIPooling, Split, SPP, Threshold, Tile, Squeeze, ExpandDims, Normalize, Permute, PriorBox, DetectionOutput, Reorg, Yolo*DetectionOutput, PSROIPooling, ROIAlign, Noop, DeepCopy, StatisticsPooling, Softplus, **Pooling1D, ConvolutionDepthWise1D, Deconvolution1D, DeconvolutionDepthWise1D**, Convolution3D/DepthWise3D, Pooling3D, Deconvolution3D/DepthWise3D, **Einsum, GLU**, DeformableConv2D, Fold, Unfold, **GridSample**, CumulativeSum, CopyTo, Diag, CELU, Shrink, **Spectrogram, InverseSpectrogram**, Flip | + +Implications for TTS/VLA: `Embed`, `GLU`, depthwise/transposed 1-D convs (common in vocoders), `Spectrogram`, and RNNs always run on CPU — on a Vulkan graph each such layer forces a download + `submit_and_wait` ([src/net.cpp#L237](../../ncnn/src/net.cpp#L237)). `SDPA`, `MultiHeadAttention`, `Gemm`, `RMSNorm`, `LayerNorm`, `RotaryEmbed`, `Convolution1D`, `Deconvolution` are covered on both ARM and Vulkan. [Verified] + +### Precision / performance option matrix [Verified] +Defaults from [src/option.cpp](../../ncnn/src/option.cpp); effects from [src/net.cpp](../../ncnn/src/net.cpp) and layer `support_*` checks. + +| Option | Default | CPU effect | Vulkan effect | +|---|---|---|---| +| `use_fp16_storage` / `use_fp16_arithmetic` / `use_fp16_packed` | true | ARMv8.2 (`asimdhp`), ARMv7 VFPv4 storage, RISC-V Zfh/Zvfh: activations cast to fp16 between layers that `support_fp16_storage` | fp16 SSBO storage, fp16 math, packed fp16x4; cleared if device lacks support | +| `use_bf16_storage` | false | Any ARM/x86/LoongArch with `NCNN_BF16`: bf16 activations (takes precedence over fp16 when set, [src/net.cpp#L400](../../ncnn/src/net.cpp#L400)) | `use_bf16_packed/storage` need `VK_KHR_shader_bfloat16` | +| `use_int8_inference` (+`use_int8_*`) | true | Runs `forward_int8` on layers carrying int8 scales | int8 storage/arithmetic shaders where present | +| `use_packing_layout` | true | elempack 4/8/16 SIMD layout negotiated per layer | elempack 4 (`support_vulkan_packing`) | +| `use_winograd_convolution` / `use_sgemm_convolution` (+`winograd23/43/63`) | true | Conv algorithm choice at `create_pipeline` (more memory) | Winograd/GEMM shaders incl. cooperative-matrix `_cm` variants | +| `use_cooperative_matrix`, `use_subgroup_ops`, `use_shader_local_memory` | true | — | Tensor-core style matmul (`VK_KHR_cooperative_matrix`/NV), subgroup reductions, shared memory (discrete GPUs only) | +| `lightmode` | true | Free intermediates early, inplace when refcount==1 | same | +| `use_local_pool_allocator` | true | Net-owned pools when app passes none | — | +| `num_threads`, `openmp_blocktime` (20 ms), `flush_denormals` (3=DAZ+FTZ) | big-core count | OpenMP width, spin time, denormal handling around each extract | CPU-side upload/cast still threaded | +| `kvcache_allocator`, `kvcache_max_seqlen_hint`, `kvcache_vkallocator` | null / 0 | KV cache placement and initial reservation | device-resident cache | +| `use_mapped_model_loading`, `use_weights_in_host_memory` | false | mmap `.bin` | keep GPU weights in host-visible memory (iGPU/UMA) | +| `vulkan_device_index` | -1 (auto by `rough_score`) | — | device choice | + +### Fallback when op unsupported [Verified] +- Unknown layer type in param → `load_param` fails ("layer %s not exists or registered", [src/net.cpp#L1417](../../ncnn/src/net.cpp#L1417)); the app must register it via `register_custom_layer` first (Piper example does this). +- Vulkan-unsupported layer/param → transparent CPU execution with upload/download around it ([src/net.cpp#L215](../../ncnn/src/net.cpp#L215); [FAQ-ncnn-vulkan.md](../../ncnn/docs/how-to-use-and-FAQ/FAQ-ncnn-vulkan.md) "layers without vulkan support"). +- Arch kernel missing → naive `src/layer/.cpp` is compiled and registered in its place ([cmake/ncnn_add_layer.cmake#L39](../../ncnn/cmake/ncnn_add_layer.cmake#L39)). +- Precision unsupported by a layer → Net converts inputs to fp32/elempack 1 for it (`support_*` false). + +### Extension points [Verified] +- Custom op: subclass `ncnn::Layer`, implement `load_param/load_model/forward(_inplace)`, set `one_blob_only/support_inplace/support_packing…`, then `net.register_custom_layer("Type", creator, destroyer, userdata)` (string or index) — tutorial in [how-to-implement-custom-layer-step-by-step.md](../../ncnn/docs/developer-guide/how-to-implement-custom-layer-step-by-step.md), [add-custom-layer.zh.md](../../ncnn/docs/developer-guide/add-custom-layer.zh.md); pnnx exports unknown modules as `moduleop` so names round-trip ([tools/pnnx/README.md](../../ncnn/tools/pnnx/README.md)). Python can register custom layers too ([python/src/main.cpp#L1046](../../ncnn/python/src/main.cpp#L1046)). +- Overwriting a builtin op with your own implementation uses the same API (`create_overwrite_builtin_layer`, [src/net.cpp#L1215](../../ncnn/src/net.cpp#L1215)). +- Custom Vulkan op: add shader, `upload_model`, `create_pipeline`, `forward(VkMat…)` ([vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md) "add vulkan compute support to layer"; GLSL helper macros in [docs/developer-guide/glsl-extension.md](../../ncnn/docs/developer-guide/glsl-extension.md)). +- Custom allocator: subclass `Allocator`/`VkAllocator` ([custom-allocator.md](../../ncnn/docs/developer-guide/custom-allocator.md)). +- Low-level op API: create/forward a single layer without a Net ([low-level-operation-api.md](../../ncnn/docs/developer-guide/low-level-operation-api.md)). +- New backend (e.g. NPU): **no plugin interface**; you would add a third `layer_xxx` slot to `Layer_final` and a third registry, mirroring how Vulkan is wired in [src/layer.cpp#L201](../../ncnn/src/layer.cpp#L201) and [src/net.cpp](../../ncnn/src/net.cpp). [Inferred] + +--- + +## 4. Model runner + +### Formats & conversion pipeline [Verified] +- Runtime format: `model.param` (text: magic `7767517`, `layer_count blob_count`, then `Type name in out blobs... k=v`) + `model.bin` (concatenated 32-bit-aligned weight blobs with optional type flag) ([param-and-model-file-structure.md](../../ncnn/docs/developer-guide/param-and-model-file-structure.md)). Binary param (`.param.bin`) + `ncnn2mem` header for string-less builds ([tools/ncnn2mem.cpp](../../ncnn/tools/ncnn2mem.cpp)); loading table in [ncnn-load-model.md](../../ncnn/docs/how-to-use-and-FAQ/ncnn-load-model.md). +- Recommended path: PyTorch → `pip install pnnx` → `pnnx.export(model, "m.pt", (x,))` → `m.ncnn.param/bin` + `m_ncnn.py` ([README](../../ncnn/README.md)). pnnx also loads ONNX and TNN ([tools/pnnx/src/load_onnx.cpp](../../ncnn/tools/pnnx/src/load_onnx.cpp), [load_tnn.cpp](../../ncnn/tools/pnnx/src/load_tnn.cpp)); 253 ncnn lowering passes in [tools/pnnx/src/pass_ncnn](../../ncnn/tools/pnnx/src/pass_ncnn) including `F_scaled_dot_product_attention.cpp`, `nn_MultiheadAttention.cpp`, `nn_RMSNorm.cpp`, `fuse_convert_rotaryembed.cpp`, `torch_stft.cpp`, `torchaudio_F_spectrogram.cpp`. CLI options `inputshape`, `inputshape2` (dynamic dim discovery), `fp16`, `optlevel`, `moduleop`, `customop` ([tools/pnnx/README.md#L85](../../ncnn/tools/pnnx/README.md#L85)). +- Legacy converters: [tools/onnx/onnx2ncnn.cpp](../../ncnn/tools/onnx/onnx2ncnn.cpp), [tools/caffe](../../ncnn/tools/caffe), [tools/darknet](../../ncnn/tools/darknet), [tools/mxnet](../../ncnn/tools/mxnet), [tools/keras](../../ncnn/tools/keras), [tools/tensorflow](../../ncnn/tools/tensorflow), [tools/mlir](../../ncnn/tools/mlir). Post-processing: [tools/ncnnoptimize.cpp](../../ncnn/tools/ncnnoptimize.cpp) (fusion, dead-op removal, fp16/fp32 storage choice), [tools/ncnnmerge.cpp](../../ncnn/tools/ncnnmerge.cpp). +- LLM decoders: export with sequence length 1, then **patch the param** to add `cache_k_in/v_in/k_out/v_out` blobs + `7=1` on every MHA/SDPA and set `Gemm 7=0` (dynamic M) — reference Python in [kvcache.md](../../ncnn/docs/developer-guide/kvcache.md); `benchncnn_llm` further rewrites `Gemm 18=` in memory for block-quant variants ([benchmark/benchncnn_llm.cpp#L72](../../ncnn/benchmark/benchncnn_llm.cpp#L72)). + +### Execution lifecycle [Verified] +1. **Init**: `Net net; net.opt.* = …; net.load_param(path|mem|AAsset|DataReader); net.load_model(...)`. `load_param` creates layers and resolves blob shape hints; `load_model` reads weights, calls `Layer::create_pipeline(opt)` (weight repacking, winograd transform, GPU pipeline creation) and `upload_model` for Vulkan ([src/net.cpp#L2022–2170](../../ncnn/src/net.cpp#L2022)). Option changes must precede loading (comments in [src/option.h](../../ncnn/src/option.h)). +2. **Warmup**: no explicit API; first `extract` triggers pool allocator growth and (Vulkan) SPIR-V compile/pipeline creation unless a `PipelineCache` file is preloaded. `benchncnn` uses 8 warmup loops by convention ([benchmark/benchncnn_llm.cpp#L42](../../ncnn/benchmark/benchncnn_llm.cpp#L42)). +3. **Step**: `Extractor ex = net.create_extractor(); ex.input(name|idx, Mat); ex.extract(name|idx, out[, type])`. `Extractor` copies `net.opt` at creation; `set_light_mode`, `set_blob_allocator`, `set_workspace_allocator`, `set_kvcache_allocator`, `set_kvcache_max_seqlen_hint`, `set_*_vkallocator` per extractor ([src/net.h#L167–256](../../ncnn/src/net.h#L167)). Light mode (default) releases intermediate blobs as soon as consumed and does inplace ops when the refcount is 1 ([src/net.cpp#L644](../../ncnn/src/net.cpp#L644)). Extract also sets OpenMP blocktime and denormal flags around the run ([src/net.cpp#L2896](../../ncnn/src/net.cpp#L2896)). +4. **Teardown**: `Extractor::clear`, `Net::clear` (destroys pipelines, frees allocators), `ncnn::destroy_gpu_instance()`. + +### Dynamic shapes [Verified] +- Tensors carry shape at runtime; layers derive output shapes in `forward`. Param shape hints (`30=` on Input/any layer) only feed `create_pipeline` heuristics ([src/net.cpp#L1488](../../ncnn/src/net.cpp#L1488)); they are not enforced. +- `Reshape 6="expr"` and `Slice` accept expressions over input dims (`0w`, `1c`, `+`, `*`, `max`, …) resolved on the host at forward time, avoiding shape-op subgraphs ([docs/developer-guide/expression.md](../../ncnn/docs/developer-guide/expression.md), [src/expression.h](../../ncnn/src/expression.h)). +- Gemm `7=` constant-M flag must be 0 for variable sequence length; MHA/SDPA accept arbitrary `seqlen` (`Mat::h`). `Input` layers accept any size; Piper feeds a variable-length phoneme id vector and gets variable-length audio ([examples/piper.cpp#L529](../../ncnn/examples/piper.cpp#L529)). +- Batch: `Mat::n` (NCNN_BATCH ON by default); the Net loops layers that lack `support_batch` per batch element and validates consistent `n` across inputs ([src/net.cpp#L662–712](../../ncnn/src/net.cpp#L662), [#L789](../../ncnn/src/net.cpp#L789)). Test: [tests/test_mat_batch.cpp](../../ncnn/tests/test_mat_batch.cpp). + +### State & KV cache [Verified] +- **Allocation & layout**: cache is a `Mat(head_dim, seqlen, num_kv_head)` whose `cstep` reserves extra sequence capacity; `Mat::h` is the logical length. `SDPA::kvcache_capacity` reserves `max_seqlen_hint` on first allocation, else `max(16, …)` up to +256, then grows geometrically by ×1.5; `create_or_grow_kvcache` reuses in place if `new_seqlen <= capacity` and the cache came from the kv allocator, otherwise allocates and copies valid history per head ([src/layer/sdpa.cpp#L28–92](../../ncnn/src/layer/sdpa.cpp#L28); same in [src/layer/multiheadattention.cpp#L230](../../ncnn/src/layer/multiheadattention.cpp#L230)). Layout is backend-private (head-contiguous on CPU, device-resident `VkMat` on Vulkan); Vulkan uses `sdpa_kvcache_append.comp` / `sdpa_kvcache_copy.comp` shaders ([src/layer/vulkan/shader/sdpa_kvcache_append.comp](../../ncnn/src/layer/vulkan/shader/sdpa_kvcache_append.comp)). +- **Paging / eviction**: none. The cache is a contiguous per-layer buffer; no paged attention, no sliding window, no eviction policy. Growth = realloc + memcpy. [Verified] +- **Reuse across calls**: consume-and-replace: `ex.input(cache_in_idx, cache); cache.release(); ex.extract(cache_out_idx, cache, 1)`; `type=1` preserves fp16/bf16/packing/allocator/capacity. A session-owned `UnlockedPoolAllocator` with `size_compare_ratio=0` (or `VkAllocator`) must be set on every Extractor of the session and differ from the blob allocator; shallow `Mat` copies are not snapshots, so beam search must clone ([kvcache.md](../../ncnn/docs/developer-guide/kvcache.md), [src/net.cpp#L2876](../../ncnn/src/net.cpp#L2876)). Whisper's beam search keeps a `std::vector kvcache` per beam ([examples/whisper.cpp#L291](../../ncnn/examples/whisper.cpp#L291)). +- **Cross-attention**: MHA with separate q and k/v inputs reuses a static cache without appending (encoder–decoder), SDPA always appends ([kvcache.md](../../ncnn/docs/developer-guide/kvcache.md) §2). +- **Attention mask / RoPE** are model inputs: `benchncnn_llm` builds a causal `attention_mask (dst_seqlen × cur_seqlen)` with −10000 and `cos/sin` caches on the host per step ([benchmark/benchncnn_llm.cpp#L285–309](../../ncnn/benchmark/benchncnn_llm.cpp#L285)); `RotaryEmbed` takes `(x, cos_cache, sin_cache)` ([src/layer/rotaryembed.cpp#L19](../../ncnn/src/layer/rotaryembed.cpp#L19)). +- **Other recurrent state**: LSTM/GRU/RNN take/return hidden-state blobs; RVM example threads 4 recurrent tensors between frames ([examples/rvm.cpp#L197](../../ncnn/examples/rvm.cpp#L197)). +- Tests: [tests/test_sdpa_kvcache.cpp](../../ncnn/tests/test_sdpa_kvcache.cpp), [tests/test_sdpa_kvcache_session.cpp](../../ncnn/tests/test_sdpa_kvcache_session.cpp), [tests/test_multiheadattention_kvcache.cpp](../../ncnn/tests/test_multiheadattention_kvcache.cpp), [tests/test_multiheadattention_kvcache_allocator.cpp](../../ncnn/tests/test_multiheadattention_kvcache_allocator.cpp); perf: [tests/perf/perf_sdpa_kvcache.cpp](../../ncnn/tests/perf/perf_sdpa_kvcache.cpp), [perf_sdpa_decode.cpp](../../ncnn/tests/perf/perf_sdpa_decode.cpp), [perf_sdpa_prefill.cpp](../../ncnn/tests/perf/perf_sdpa_prefill.cpp). + +### Multimodal stages [Verified] +- ncnn has no notion of stages; each stage is a separate `Net`, glued by host code. Whisper: `fbank` net (waveform → log-mel, uses Spectrogram-based graph [Inferred]) → `encoder` → `embed_token` + `embed_position` → `decoder` (self-attn with dynamic cache, cross-attn with static encoder k/v) → `proj_out` ([examples/whisper.cpp#L310–318](../../ncnn/examples/whisper.cpp#L310)). Piper: `enc_p` → `emb_g` (speaker) → `dp` (duration predictor with rational-quadratic spline custom layer) → host `path_attention` (length regulation + noise) → `flow` → `dec` (HiFi-GAN-style vocoder inside VITS) → PCM ([examples/piper.cpp#L518–675](../../ncnn/examples/piper.cpp#L518)). +- Vision: `Mat::from_pixels_resize` + `substract_mean_normalize` → detector net → host NMS/decode (e.g. [examples/yolo11.cpp#L352](../../ncnn/examples/yolo11.cpp#L352)); Vulkan zero-copy camera path via AHB import ([docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md](../../ncnn/docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md)). +- Diffusion: no sampler/scheduler code in tree. [Verified by absence: grep for "diffusion|ddim|scheduler" in src/examples returns nothing relevant] + +--- + +## 5. Scheduler + +### (a) Request / token / pipeline-level scheduling — **N/A** [Verified] +- There is no request queue, no batch former, no continuous batching, no prefill/decode scheduler, no multi-stage pipeline orchestrator. The unit of work is one synchronous `Extractor::extract` call on the calling thread. Prefill vs decode is purely the host choosing `seqlen>1` vs `seqlen=1` inputs against the same Net ([benchmark/benchncnn_llm.cpp#L406–469](../../ncnn/benchmark/benchncnn_llm.cpp#L406), [kvcache.md](../../ncnn/docs/developer-guide/kvcache.md) §5). +- Concurrency model: a `Net` is immutable after load and can be shared by many threads, each creating its own `Extractor` (documented pattern with per-thread `UnlockedPoolAllocator` + shared locked workspace allocator, [custom-allocator.md](../../ncnn/docs/developer-guide/custom-allocator.md); Vulkan "batch inference" with `omp parallel for` over extractors bounded by `compute_queue_count()`, [vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md)). An `Extractor` itself is single-threaded state (`blob_mats` vector) and not safe to share. [Inferred] +- Cancellation/abort: none; a running `extract` cannot be interrupted. Backpressure: none (host's job). Timeouts: only the Vulkan chunked-submit heuristic to avoid GPU driver watchdogs ([src/net.cpp#L250](../../ncnn/src/net.cpp#L250)). +- Batching across requests: `NCNN_BATCH` allows `n>1` in a single extract but the KV cache rejects `n>1` ([src/net.cpp#L2882](../../ncnn/src/net.cpp#L2882)), so LLM decode is strictly one sequence per extract. + +### (b) Graph / operator / thread-level scheduling [Verified] +- **Op ordering**: depth-first recursion from the requested blob to its producers (`forward_layer` recursion over `layer->bottoms`), i.e. topological order induced by param order; memoized in `blob_mats` so shared subgraphs run once per Extractor ([src/net.cpp#L123–150](../../ncnn/src/net.cpp#L123)). Multiple `extract` calls on one Extractor reuse computed blobs (e.g. Piper extracts `out0/out1/out2` from one run). +- **Intra-op threading**: OpenMP `parallel for` inside kernels with `num_threads(opt.num_threads)`; per-layer `featmask` bit 7 forces single thread; `set_omp_num_threads`, `set_kmp_blocktime` (default 20 ms spin), `set_cpu_powersave(0|1|2)` binds to all/little/big cores (Android/Linux), `set_cpu_thread_affinity(CpuSet)` ([src/cpu.h#L145–170](../../ncnn/src/cpu.h#L145); [openmp-best-practice.md](../../ncnn/docs/how-to-use-and-FAQ/openmp-best-practice.md)). `NCNN_SIMPLEOMP` provides a minimal LLVM-ABI OpenMP runtime supporting only `parallel for num_threads` ([src/simpleomp.h](../../ncnn/src/simpleomp.h)); `NCNN_THREADS=OFF` builds single-threaded (WASM CI does this). +- **Inter-op parallelism**: none on CPU (sequential layer execution). On Vulkan, layers are recorded back-to-back into one command buffer with automatic barriers; no multi-queue/multi-stream overlap inside one extract. Overlap is achievable only by running several Extractors on several threads/queues. +- **Stream assignment**: one `VkCompute` (compute queue) per extract; `VkTransfer` (transfer queue if `unified_compute_transfer_queue()` false) for weight upload ([src/gpu.h#L257–264](../../ncnn/src/gpu.h#L257)). +- **Memory scheduling**: light mode frees blobs after last use; pool allocators recycle; no static memory planning. +- **A53/A55 tuning**: `use_a53_a55_optimized_kernel` auto-set from the current thread's core ([src/option.cpp](../../ncnn/src/option.cpp), [docs/developer-guide/arm-a53-a55-dual-issue.md](../../ncnn/docs/developer-guide/arm-a53-a55-dual-issue.md)). + +--- + +## 6. I/O layer + +- **Image preprocessing** [Verified]: `Mat::from_pixels / from_pixels_resize / from_pixels_roi / from_pixels_roi_resize` with `PIXEL_RGB/BGR/GRAY/RGBA/BGRA` and `X2Y` conversions, stride support, `to_pixels*`, `substract_mean_normalize`, `from_android_bitmap*` ([src/mat.h#L255–335](../../ncnn/src/mat.h#L255)); resize/rotate/affine/drawing helpers ([src/mat_pixel_resize.cpp](../../ncnn/src/mat_pixel_resize.cpp), [src/mat_pixel_rotate.cpp](../../ncnn/src/mat_pixel_rotate.cpp), [src/mat_pixel_affine.cpp](../../ncnn/src/mat_pixel_affine.cpp), [src/mat_pixel_drawing.cpp](../../ncnn/src/mat_pixel_drawing.cpp); doc [efficient-roi-resize-rotate.md](../../ncnn/docs/how-to-use-and-FAQ/efficient-roi-resize-rotate.md)). `NCNN_SIMPLEOCV` supplies `cv::Mat/imread/imwrite` stand-ins ([src/simpleocv.h](../../ncnn/src/simpleocv.h)) backed by [src/stb_image.h](../../ncnn/src/stb_image.h). Vulkan YCbCr camera import shader [src/convert_ycbcr.comp](../../ncnn/src/convert_ycbcr.comp). +- **Audio preprocessing** [Verified]: no audio I/O library. `Spectrogram` (n_fft, hop, win, hann/hamming, center/pad, power, onesided, normalized) and `InverseSpectrogram` layers implement STFT/iSTFT in-graph with a naive DFT (cos/sin loops) ([src/layer/spectrogram.cpp](../../ncnn/src/layer/spectrogram.cpp), [src/layer/inversespectrogram.cpp](../../ncnn/src/layer/inversespectrogram.cpp)); Whisper reads 16-bit WAV by hand (`load_wav_samples`, [examples/whisper.cpp#L829](../../ncnn/examples/whisper.cpp#L829)) and Piper writes a WAV header by hand ([examples/piper.cpp#L677](../../ncnn/examples/piper.cpp#L677)). +- **Tokenization** [Verified]: none in the library. Whisper example: plain reverse-vocab list (`whisper_vocab.txt`) with hard-coded special-token ids ([examples/whisper.cpp#L31–200](../../ncnn/examples/whisper.cpp#L31)); Piper: binary word→phoneme-id dictionary + rule-based sentence splitting (`simple_phonemize`, [examples/piper.cpp#L323](../../ncnn/examples/piper.cpp#L323)); `benchncnn_llm` feeds random embeddings (no tokenizer). `Embed` layer maps ids→vectors in-graph ([src/layer/embed.h](../../ncnn/src/layer/embed.h)). +- **Streaming** [Verified]: no streaming API. Token streaming falls out naturally from the per-step decode loop (the app owns the loop). Audio streaming/chunking is not provided; Piper synthesizes the whole utterance then normalizes/clips to int16 ([examples/piper.cpp#L646](../../ncnn/examples/piper.cpp#L646)). Chunked/streaming vocoding would need the model to be exported in a chunkable form. [Inferred] +- **Buffering** [Verified]: `Mat` refcounted buffers; `Extractor::input` stores a shallow reference (no copy) ([src/net.cpp#L2861](../../ncnn/src/net.cpp#L2861)); external memory `Mat(w,h,c,data)` constructors allow zero-copy input. +- **Postprocessing** [Verified]: host code in examples (argmax/greedy in whisper `run_decoder_*`, beam search with log-softmax, NMS in detectors). No sampling library (temperature/top-p) in tree. +- **Application boundaries** [Verified]: + - C++ API: [src/net.h](../../ncnn/src/net.h), [src/mat.h](../../ncnn/src/mat.h), [src/option.h](../../ncnn/src/option.h). + - C API (`NCNN_C_API`, default ON): [src/c_api.h](../../ncnn/src/c_api.h) incl. `ncnn_mat_create_*_batch`, `ncnn_option_set_kvcache_*`, `ncnn_pipelinecache_*`. + - Python (`pip install ncnn`, pybind11): `ncnn.Net/Extractor/Mat/Option/Layer/PoolAllocator/VulkanDevice…`, `extract(name, type)` returning `(ret, Mat)`, kv-cache setters ([python/src/main.cpp#L959](../../ncnn/python/src/main.cpp#L959), [python/README.md](../../ncnn/python/README.md)); a model zoo of detection/pose/classification wrappers ([python/ncnn/model_zoo](../../ncnn/python/ncnn/model_zoo)). + - Android: `AAssetManager` loaders ([src/net.h#L110](../../ncnn/src/net.h#L110)), `Mat::from_android_bitmap`, AHB Vulkan import; the JNI demo project moved out of tree ([examples/squeezencnn/README.md](../../ncnn/examples/squeezencnn/README.md)). + - iOS: `Info.plist` + `ios.toolchain.cmake`; MoltenVK for GPU. + - HTTP server / CLI serving: none. Only benchmark/example binaries ([benchmark/benchncnn.cpp](../../ncnn/benchmark/benchncnn.cpp)). + +--- + +## 7. Execution flow + +### (a) Text LLM decode with KV cache (as in `benchncnn_llm` / kvcache.md) [Verified] + +```mermaid +sequenceDiagram + participant App + participant Net as Net(decoder) + participant Ex as Extractor + participant NP as NetPrivate::forward_layer + participant SDPA as SDPA layer (per block) + participant Gemm as Gemm/RMSNorm/RoPE layers + participant Proj as Net(proj_out) + + App->>Net: load_param(decoder.param patched 7=1, Gemm 7=0) ; load_model(bin) + Note over Net: create_pipeline per layer, (Vulkan: upload weights, compile SPIR-V) + App->>App: prefill: tokens -> embeds Mat(hidden, seqlen), causal mask, cos/sin cache + App->>Ex: create_extractor(); set_kvcache_allocator(pool); set_kvcache_max_seqlen_hint(N) + App->>Ex: input(in0..in3), input(cache_k_in_i, empty) for all i + App->>Ex: extract(cache_k_out_i, cache[i], type=1) for all i + Ex->>NP: forward_layer(producer of cache_k_out_i) + NP->>Gemm: RMSNorm -> Gemm(q,k,v) -> RotaryEmbed + NP->>SDPA: forward(q,k,v,mask,past_k,past_v) + SDPA->>SDPA: create_or_grow_kvcache (reserve hint) ; append ; flash-attn QK^T softmax V + SDPA-->>NP: out, cache_k_out, cache_v_out (private layout) + NP-->>Ex: blob_mats filled (memoized) + App->>Ex: extract("out0", hidden) (already computed for last block, no re-run) + App->>Proj: Extractor.input(last row).extract("out0", logits) + App->>App: argmax/sample -> next token + loop decode step + App->>Ex: new Extractor; input(embed of 1 token, mask (past+1 x 1), cos/sin row) + App->>Ex: input(cache_k_in_i, cache[i]); cache[i].release() + App->>Ex: extract(cache_k_out_i, cache[i], 1) ; extract("out0") + Note over SDPA: cache reused in place when seqlen <= capacity, else grow x1.5 + memcpy + App->>Proj: logits -> next token + end +``` + +### (b) TTS / ASR path — Whisper ASR (encoder–decoder with cross-attention cache) and Piper TTS [Verified] + +```mermaid +sequenceDiagram + participant App as whisper.cpp + participant FB as Net(fbank) + participant Enc as Net(encoder) + participant Emb as Net(embed_token/position) + participant Dec as Net(decoder, MHA 7=1) + participant PO as Net(proj_out) + + App->>App: load_wav_samples(16-bit PCM) -> Mat waveform + App->>FB: Extractor.input("in0", waveform).extract("out0", input_features) + Note over FB: log-mel via Spectrogram-based graph [Inferred] + App->>Enc: input(input_features).extract("out0", encoder_states) + App->>App: prompt ids (SOT, lang, task, no_timestamps) + App->>Emb: ids -> token_embeds ; positions -> position_embeds ; sum + App->>Dec: input(in0=embeds, in1=encoder_states, in2=attention_mask) + App->>Dec: extract(cache_k/v_out_i, kv[i], type=1) for all MHA ; extract("out0") + Note over Dec: self-attn MHA appends to dynamic cache; cross-attn MHA uses static k/v from encoder_states + App->>PO: last state -> logits ; log_softmax ; beam search candidates keep own kv vector + loop until EOT + App->>Dec: input(1 token embeds, encoder_states, mask, kv[i]) ; extract(kv_out, "out0") + App->>PO: logits -> beam update + end + App->>App: detokenize via reverse vocab -> text + + Note over App: Piper TTS (examples/piper.cpp): text -> simple_phonemize (dict) -> Net enc_p (custom relative_embeddings layers) -> Net emb_g(speaker) -> Net dp (custom spline layer, noise) -> host path_attention (durations, noise) -> Net flow -> Net dec (vocoder) -> normalize -> int16 PCM -> WAV 22050 Hz +``` + +--- + +## 8. Tests and examples + +- **Unit tests** [Verified]: 181 `tests/test_*.cpp` (per-layer randomized tests comparing naive vs optimized vs Vulkan via [tests/testutil.h](../../ncnn/tests/testutil.h); `_oom` variants exercise allocation failure). Build with `-DNCNN_BUILD_TESTS=ON`; each becomes a CTest via [cmake/run_test.cmake](../../ncnn/cmake/run_test.cmake) ([tests/CMakeLists.txt](../../ncnn/tests/CMakeLists.txt)). Non-layer tests: `test_c_api`, `test_cpu`, `test_expression`, `test_modelbin`, `test_paramdict`, `test_mat_batch`, `test_mat_pixel*`, `test_squeezenet`, and with Vulkan `test_command`, `test_pipeline_cache`. Transformer-specific: `test_sdpa*`, `test_multiheadattention*` (incl. `_kvcache`, `_block_quant`), `test_rotaryembed`, `test_rmsnorm`, `test_gemm_*` (incl. `_block_quant`, `_nt_int8`), `test_embed`, `test_spectrogram`, `test_inversespectrogram`, `test_glu`, `test_lstm/gru`. +- **How to run (per repo docs/CI)** [Verified]: + ```shell + # library + tests (CPU) + mkdir build && cd build + cmake -DNCNN_BUILD_TESTS=ON .. # add -DNCNN_VULKAN=ON for GPU tests (needs glslang submodule) + cmake --build . -j $(nproc) + ctest --output-on-failure -j $(nproc) # exact invocation used by CI: .github/workflows/linux-x64-cpu-gcc.yml + # model benchmark (CPU, 4 big-core threads, 4 loops, cooldown) / GPU device 0 + ./benchmark/benchncnn 4 4 2 -1 1 + ./benchmark/benchncnn 8 1 2 0 0 + # LLM prefill/decode tokens-per-second table + ./benchmark/benchncnn_llm [loop count] [num threads] [powersave] [gpu device] [cooling down] + # examples (need OpenCV or -DNCNN_SIMPLEOCV=ON) — piper needs en_*.ncnn.param/bin + en-word_id.bin in cwd + ./examples/piper "Hello World" 0 out.wav + ./examples/whisper ... # needs whisper_vocab.txt and exported nets + ``` + Sources: [docs/how-to-build/how-to-build.md#L40](../../ncnn/docs/how-to-build/how-to-build.md#L40), [.github/workflows/linux-x64-cpu-gcc.yml#L53](../../ncnn/.github/workflows/linux-x64-cpu-gcc.yml#L53), [benchmark/README.md](../../ncnn/benchmark/README.md), [examples/piper.cpp#L718](../../ncnn/examples/piper.cpp#L718). Note this checkout cannot build Vulkan or Python until `git submodule update --init` (glslang, pybind11). +- **Perf micro-benchmarks** [Verified]: [tests/perf](../../ncnn/tests/perf) (`perf_sdpa_prefill/decode/kvcache`, conv, gemm-ish ops). +- **Model benchmarks** [Verified]: `benchncnn` (36 built-in params incl. `vision_transformer.param`, int8 variants; usage `./benchncnn [loops] [threads] [powersave] [gpu] [cooldown] param=… shape=…`) and `benchncnn_llm` (7 LLM decoders, prints prefill/decode tokens/s; `[loop] [threads] [powersave] [gpu] [cooldown]`) ([benchmark/README.md](../../ncnn/benchmark/README.md), [benchmark/benchncnn_llm.cpp#L643](../../ncnn/benchmark/benchncnn_llm.cpp#L643)). README carries repo-reported timing tables for many boards (Jetson AGX Orin/Orin Nano/Nano/TX2 NX, RK3588, Snapdragon 8xx/X Elite, Raspberry Pi 3/4/5, Apple A7, Loongson…) — repo-reported, not reproduced. +- **Examples** [Verified] ([examples/CMakeLists.txt](../../ncnn/examples/CMakeLists.txt)): vision — squeezenet (+C API), yolov2…yolo11 (det/seg/pose/obb/cls), yoloworld, yolox, nanodet, scrfd, retinaface, arcface, simplepose, yolact, rvm (video matting w/ recurrent state), ppocrv5 (OCR), p2pnet, mobilenet/squeezenet SSD; audio — [piper.cpp](../../ncnn/examples/piper.cpp) (TTS), [whisper.cpp](../../ncnn/examples/whisper.cpp) (ASR). Python examples mirror detection models ([python/examples](../../ncnn/python/examples)). Mobile app demos are external (nihui/ncnn-android-*), only a README remains in [examples/squeezencnn](../../ncnn/examples/squeezencnn). +- **CI** [Verified]: 45 workflows in [.github/workflows](../../ncnn/.github/workflows): per-OS/arch CPU & GPU (Linux x64 GPU via SwiftShader/lavapipe [Inferred]), Android (armeabi-v7a/arm64-v8a/x86/x86_64/riscv64, Vulkan ON), iOS/macOS/Catalyst/tvOS/visionOS/watchOS, HarmonyOS, WebAssembly (3 variants incl. simpleomp threads), ESP32, RISC-V ELF, Windows (MSVC/clang/mingw/XP/ARM), Intel SDE for AVX-512 variants, code-format, CodeQL, coverage, `compare-binary-size` (reports `libncnn.so` delta for x86_64/armhf/aarch64 on PRs), pnnx and python wheel builds. + +--- + +## 9. TTS relevance + +**What exists** [Verified] +- End-to-end TTS example: Piper/VITS split into 5 nets with 3 app-registered custom layers, speaker embedding, duration predictor, flow, and neural vocoder (inside `dec`), 22.05 kHz int16 output; Vulkan enabled per net (`opt.use_vulkan_compute = true`) ([examples/piper.cpp](../../ncnn/examples/piper.cpp)). Export recipe references nihui/ncnn-android-piper `export_ncnn.py` (external). +- ASR counterpart with the full encoder–decoder + KV-cache machinery (Whisper tiny…large-v3-turbo) ([examples/whisper.cpp](../../ncnn/examples/whisper.cpp)). +- Audio ops: `Spectrogram`/`InverseSpectrogram` (STFT/iSTFT; pnnx maps `torch.stft/istft`, `torchaudio.functional.spectrogram/inverse_spectrogram` — [tools/pnnx/src/pass_ncnn/torch_stft.cpp](../../ncnn/tools/pnnx/src/pass_ncnn/torch_stft.cpp), [torchaudio_F_inverse_spectrogram.cpp](../../ncnn/tools/pnnx/src/pass_ncnn/torchaudio_F_inverse_spectrogram.cpp)), `Convolution1D/ConvolutionDepthWise1D/Deconvolution1D/DeconvolutionDepthWise1D` (with ARM fp16 and Vulkan kernels: [src/layer/arm/convolution1d_arm_asimdhp.cpp](../../ncnn/src/layer/arm/convolution1d_arm_asimdhp.cpp), [src/layer/vulkan/convolution1d_vulkan.cpp](../../ncnn/src/layer/vulkan/convolution1d_vulkan.cpp)), `Pooling1D`, `LSTM/GRU`, `LayerNorm/GroupNorm/InstanceNorm`, `Gemm/MatMul`, `MultiHeadAttention/SDPA` with cache — sufficient for VITS, HiFi-GAN/Vocos-style vocoders, Tacotron-like RNNs and transformer TTS decoders. +- Transformer-decoder TTS (e.g. token-based LLM-TTS such as CosyVoice/Fish-Speech-style AR heads) can reuse the LLM decode loop with `SDPA 7=1`; speech-token → mel/flow-matching and vocoder stages would be additional nets. [Inferred] + +**What is missing** [Verified unless noted] +- No tokenizer/G2P/phonemizer library (only an English dictionary hack in the example; comment says "works for english only", [examples/piper.cpp#L325](../../ncnn/examples/piper.cpp#L325)). +- No audio I/O, resampling, mel filterbank utility, or vocoder library; no streaming/chunked synthesis API; no FFT acceleration (Spectrogram is a naive DFT [Verified from cos/sin loops], likely slow for long inputs [Inferred]). +- No sampler (temperature/top-k/top-p/repetition penalty) or beam search utility outside the Whisper example. +- No neural audio codec (EnCodec/SNAC/DAC) implementations or examples; codebook lookup can be done with `Embed` [Inferred]. +- No batched multi-utterance decode (KV cache is batch-1). +- Piper export requires patches to the upstream repo and an external script (not in tree). + +--- + +## 10. VLA relevance + +**What exists** [Verified] +- Vision encoders: CNN backbones are the historical core (winograd/sgemm/int8/fp16 conv on ARM/x86/Vulkan); ViT is exercised by `benchmark/models/vision_transformer.param` and `MultiHeadAttention/LayerNorm/GELU/Gemm` kernels ([benchmark/models/vision_transformer.param](../../ncnn/benchmark/models/vision_transformer.param)); `Interp`, `GridSample`, `DeformableConv2D`, `ROIAlign` for spatial ops. Camera zero-copy on Android via AHB. +- Structured / proprioceptive inputs: any number of `Input` blobs (1-D `Mat`), `Concat`, `Gemm/InnerProduct`, `Embed` for discrete state; expression-based reshape for variable shapes. +- Action heads: MLP (`InnerProduct`/`Gemm` + activations), Gaussian/mixture heads (host post-process), autoregressive token heads via the LLM path (`Embed` → decoder with `SDPA 7=1` → `Gemm` proj). +- Diffusion/flow policy building blocks: `Convolution1D`/`GroupNorm`/`Mish`/`SiLU`(Swish) (Diffusion Policy U-Net 1D), transformer blocks, `Embed` for timestep tables, `RotaryEmbed`. Each denoise step = one `Extractor` call with `(x_t, t_embed, cond)` inputs; host implements the DDIM/flow-matching update. [Inferred] +- Low-latency loop support: light mode + pooled allocators, `set_cpu_powersave(2)` big-core pinning, `openmp_blocktime` tuning, Vulkan `PipelineCache` to avoid first-run shader compilation, mmap model loading, `VkMat` chaining across nets without downloads ([vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md)). + +**What is missing** [Verified unless noted] +- No VLA/robotics example, no diffusion-policy or flow-matching sampler, no action chunking utilities. +- No multi-input scheduling: vision encoder and language decoder are separate Nets executed serially by the host; no async overlap primitive beyond threads + separate Extractors [Inferred]. +- No paged/shared KV cache across parallel action samples (batch-1 cache); sampling K action candidates needs K cache clones. +- No NPU/DSP delegates (Hexagon/QNN, Ethos, RKNPU, Jetson DLA/CUDA); GPU only via Vulkan, so Jetson runs CPU or Tegra-Vulkan, not TensorRT. +- Vision-language projector/token merging (e.g. SigLIP+MLP) must be exported as ordinary Gemm graphs; multimodal embedding concatenation is host code [Inferred]. + +--- + +## 11. Edge deployment profile + +- **Build system** [Verified]: CMake ≥ 2.8.12 (3.10 policies), C++03/11 compatible core (`host.gcc-c++03.toolchain.cmake` exists), no runtime third-party deps. Key options ([CMakeLists.txt#L61–110](../../ncnn/CMakeLists.txt#L61)): `NCNN_VULKAN` (OFF), `NCNN_SIMPLEVK` (ON), `NCNN_SYSTEM_GLSLANG` (OFF), `NCNN_OPENMP` (ON), `NCNN_SIMPLEOMP/SIMPLESTL/SIMPLEOCV/SIMPLEMATH` (OFF), `NCNN_THREADS` (ON), `NCNN_RUNTIME_CPU` (ON), `NCNN_INT8` (ON), `NCNN_WEIGHT_QUANT` (ON), `NCNN_BF16` (ON), `NCNN_BATCH` (ON), `NCNN_C_API` (ON), `NCNN_PLATFORM_API` (ON), `NCNN_PIXEL*` (ON), `NCNN_STDIO/STRING` (ON), `NCNN_SHARED_LIB` (OFF), `NCNN_DISABLE_RTTI/EXCEPTION` (ON when tools/examples off), `NCNN_BUILD_TOOLS/EXAMPLES/BENCHMARK/TESTS`, `NCNN_PYTHON`, per-ISA `NCNN_ARM82/ARM82DOT/ARM84BF16/ARM84I8MM/ARM86SVE*`, `NCNN_AVX2/AVX512*/AVXVNNI*`, `NCNN_RVV/ZFH/ZVFH/XTHEADVECTOR`, `NCNN_MSA/LSX/LASX`, `WITH_LAYER_`. +- **Binary size** [Verified/Unknown]: the repo does not publish absolute sizes; CI `compare-binary-size` computes `libncnn.so` deltas per PR for x86_64/armhf/aarch64 ([.github/workflows/compare-binary-size.yml](../../ncnn/.github/workflows/compare-binary-size.yml)). The size cheatsheet ([build-minimal-library.md](../../ncnn/docs/how-to-use-and-FAQ/build-minimal-library.md)) lists reductions: disable RTTI/exceptions, Vulkan, STDIO/STRING, BF16/INT8/WEIGHT_QUANT, pixel functions, OpenMP, unused ISA kernels, runtime dispatch, unused layers, and STL (`simplestl`, `-nodefaultlibs -fno-builtin -nostdinc++`). Absolute numbers: [Unknown]. +- **Dependencies** [Verified]: none at runtime. Optional: OpenMP runtime (or simpleomp), Vulkan driver (loaded dynamically; no libvulkan link with simplevk), glslang (submodule, statically linked when `NCNN_VULKAN=ON`; not initialized in this checkout), OpenCV or simpleocv for examples/tools, protobuf for legacy converters, libtorch for building pnnx from source (pip wheel avoids it). +- **Platforms** [Verified]: + - Embedded NVIDIA/Jetson: aarch64 toolchain + Tegra Vulkan ([how-to-build.md#L66](../../ncnn/docs/how-to-build/how-to-build.md#L66)); benchmark tables for AGX Orin/Orin Nano/Nano/TX2 NX (repo-reported). No CUDA/TensorRT. + - ARM CPU: ARMv7 (NEON, VFPv4 fp16), ARMv8/8.2/8.4/8.6 (fp16, dotprod, fhm, bf16, i8mm, SVE/SVE2 flags), A53/A55 dual-issue tuning, big.LITTLE affinity; toolchains for Cortex-A cross builds, HiSilicon Hi35xx, Rockchip, Allwinner, Raspberry Pi ([toolchains](../../ncnn/toolchains)). + - Apple: macOS/iOS/tvOS/visionOS/watchOS builds via [toolchains/ios.toolchain.cmake](../../ncnn/toolchains/ios.toolchain.cmake) and CI; GPU only through MoltenVK (Vulkan→Metal), no native Metal/CoreML/ANE. + - Android: NDK builds for all ABIs incl. riscv64, Vulkan on Adreno/Mali (API 24+ with system loader; simplevk allows <24), `AAssetManager` loading, AHB zero-copy (API 26+), JNI bitmap helpers. Prebuilt release archives are built with `-DANDROID_PLATFORM=android-19/21 -DNCNN_VULKAN=ON` ([build-android.cmd](../../ncnn/build-android.cmd)). No NNAPI/QNN. + - HarmonyOS (`ohos.toolchain.cmake`, `__OHOS__` mmap path), QNX, Windows XP/ARM, WebAssembly (single-thread or simpleomp+pthreads, SIMD128 via SSE2 path), ESP32/RISC-V bare metal (`riscv64-unknown-elf`, XuanTie C906/C907/C908/C910, SpacemiT K1), LoongArch, MIPS, PowerPC. +- **Model conversion path & constraints** [Verified]: PyTorch → pnnx (needs `inputshape` for shape propagation; `inputshape2` to mark dynamic dims; unsupported modules → `moduleop` + custom layer; fp16 weights default) → optional `ncnnoptimize` (skip if pnnx) → optional `ncnn2table/ncnn2int8` or `ncnnllm2table/ncnnllm2int`. ONNX also via pnnx (preferred) or `onnx2ncnn`. Constraints: batch dim removed by pnnx (`Mat` is CHW-ish, WHC ordering in tools), ops must exist among the 110 types, control flow must be unrolled or hosted, LLM decoders need param patching for cache and dynamic-M Gemm, Vulkan-unsupported layers (e.g. `Embed`, `RNN/LSTM/GRU`, `Spectrogram` — no `*_vulkan.cpp` in [src/layer/vulkan](../../ncnn/src/layer/vulkan)) fall back to CPU with sync cost. + +--- + +## 12. Limitations and unknowns + +1. [Verified] No request scheduler, batching server, or streaming API; all orchestration is application code. +2. [Verified] KV cache is per-layer contiguous with geometric growth and batch size 1; no paging, sliding window, prefix sharing or eviction. +3. [Verified] Attention mask and RoPE cos/sin tables are host-provided inputs each step; positions are implicit in the mask shape. +4. [Verified] Vulkan shaders compile at runtime (glslang); first-run latency unless a `PipelineCache` file is shipped; cache is not portable across devices/drivers/builds ([vulkan-pipeline-cache.md#L288](../../ncnn/docs/developer-guide/vulkan-pipeline-cache.md#L288)). +5. [Verified] No CUDA/Metal/CoreML/OpenCL/NNAPI/QNN backends; NPUs on phones and Jetson tensor cores are unreachable except through Vulkan cooperative-matrix shaders (15 `*_cm.comp` shaders incl. `sdpa_fa_cm.comp`, `gemm_cm.comp`). +6. [Verified] W4/W6/W8 block-quantized Gemm/MHA run on CPU only and disable fp16/bf16/packing/Vulkan for those layers. +7. [Verified] Piper/Whisper export scripts and Android demo apps live in external nihui/* repos; Piper needs a patched upstream. +8. [Verified] `Spectrogram` uses a direct DFT (no FFT library); [Inferred] cost grows as O(n_fft²) per frame. +9. [Verified] glslang and pybind11 submodules are not initialized in this checkout, so Vulkan and Python builds are not possible from it as-is. +10. [Unknown] Absolute library binary sizes per configuration (only CI deltas exist). +11. [Unknown] Real throughput of `benchncnn_llm` on target edge boards; README LLM section describes the harness but no LLM numbers were found in the tables reviewed. +12. [Unknown] Thread-safety guarantees of `VulkanDevice` allocator/queue pools under heavy multi-thread use beyond the documented `compute_queue_count()` bound. +13. [Inferred] Multi-Net pipelines (encoder → decoder) pay a host round-trip unless both are Vulkan and chained with `VkMat` + shared `VkCompute`; CPU-fallback layers inside a GPU graph force `submit_and_wait`. +14. [Inferred] Very large vocab `Gemm` proj_out on CPU dominates decode time for small LLMs; no fused sampling. +15. [Proposal] For a vLLM-Omni edge backend, wrap `Net`+`Extractor` per stage behind an executor that owns the kv-cache allocator per sequence, keeps a `PipelineCache` per device, and pins decode threads to big cores; use `extract(type=1)` handles as opaque cache objects. + +--- + +## 13. Reference index + +| Path | Role | +|---|---| +| [../../ncnn/README.md](../../ncnn/README.md) | Project overview, quick start (pnnx → ncnn), release links | +| [../../ncnn/CMakeLists.txt](../../ncnn/CMakeLists.txt) | Top-level build options and ISA feature flags | +| [../../ncnn/build-android.cmd](../../ncnn/build-android.cmd) | Flags used for prebuilt Android archives | +| [../../ncnn/src/net.h](../../ncnn/src/net.h) | `Net`/`Extractor` public API incl. kv-cache setters | +| [../../ncnn/src/net.cpp](../../ncnn/src/net.cpp) | Graph load, layer creation/fallback, lazy forward, layout conversion, batch loop, Vulkan submit heuristics | +| [../../ncnn/src/layer.h](../../ncnn/src/layer.h) | `Layer` interface, `support_*` flags, creator macros | +| [../../ncnn/src/layer.cpp](../../ncnn/src/layer.cpp) | Registry lookup, `Layer_final` CPU/Vulkan wrapper | +| [../../ncnn/src/layer_declaration.h.in](../../ncnn/src/layer_declaration.h.in) / [layer_registry.h.in](../../ncnn/src/layer_registry.h.in) / [layer_type_enum.h.in](../../ncnn/src/layer_type_enum.h.in) | Generated registry templates | +| [../../ncnn/src/mat.h](../../ncnn/src/mat.h) | `Mat`/`VkMat`/`VkImageMat`, pixel conversion API, batch dim | +| [../../ncnn/src/option.h](../../ncnn/src/option.h) / [option.cpp](../../ncnn/src/option.cpp) | Runtime options and defaults | +| [../../ncnn/src/allocator.h](../../ncnn/src/allocator.h) / [allocator.cpp](../../ncnn/src/allocator.cpp) | Host/Vulkan allocators and pools | +| [../../ncnn/src/cpu.h](../../ncnn/src/cpu.h) / [cpu.cpp](../../ncnn/src/cpu.cpp) | ISA detection, topology, affinity, OpenMP wrappers | +| [../../ncnn/src/ruapu.h](../../ncnn/src/ruapu.h) | Bundled ISA detection header | +| [../../ncnn/src/gpu.h](../../ncnn/src/gpu.h) / [gpu.cpp](../../ncnn/src/gpu.cpp) | Vulkan instance/device/GpuInfo, shader compile | +| [../../ncnn/src/simplevk.h](../../ncnn/src/simplevk.h) / [simplevk.cpp](../../ncnn/src/simplevk.cpp) | In-house Vulkan loader (dlopen, MoltenVK, Android HAL) | +| [../../ncnn/src/command.h](../../ncnn/src/command.h) / [command.cpp](../../ncnn/src/command.cpp) | `VkCompute`/`VkTransfer` recording and submission | +| [../../ncnn/src/pipeline.h](../../ncnn/src/pipeline.h) / [pipelinecache.h](../../ncnn/src/pipelinecache.h) | Vulkan pipeline objects and persistent cache | +| [../../ncnn/src/paramdict.h](../../ncnn/src/paramdict.h) / [modelbin.h](../../ncnn/src/modelbin.h) / [datareader.h](../../ncnn/src/datareader.h) | Param/weight parsing abstractions | +| [../../ncnn/src/expression.h](../../ncnn/src/expression.h) | Runtime shape expression evaluator | +| [../../ncnn/src/c_api.h](../../ncnn/src/c_api.h) | C API | +| [../../ncnn/src/simpleomp.h](../../ncnn/src/simpleomp.h) / [simplestl.h](../../ncnn/src/simplestl.h) / [simpleocv.h](../../ncnn/src/simpleocv.h) / [simplemath.h](../../ncnn/src/simplemath.h) | Minimal runtime shims | +| [../../ncnn/src/stb_image.h](../../ncnn/src/stb_image.h) | Image decoding for simpleocv | +| [../../ncnn/src/convert_ycbcr.comp](../../ncnn/src/convert_ycbcr.comp) | Camera YCbCr → RGB Vulkan shader | +| [../../ncnn/src/mat_pixel_resize.cpp](../../ncnn/src/mat_pixel_resize.cpp) / [mat_pixel_rotate.cpp](../../ncnn/src/mat_pixel_rotate.cpp) / [mat_pixel_affine.cpp](../../ncnn/src/mat_pixel_affine.cpp) / [mat_pixel_drawing.cpp](../../ncnn/src/mat_pixel_drawing.cpp) | Pixel preprocessing utilities | +| [../../ncnn/src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt) | Layer registration list, shader embedding, glslang link | +| [../../ncnn/src/layer](../../ncnn/src/layer) | Naive layer implementations (110 types) | +| [../../ncnn/src/layer/sdpa.h](../../ncnn/src/layer/sdpa.h) / [sdpa.cpp](../../ncnn/src/layer/sdpa.cpp) | SDPA with GQA and kv cache growth logic | +| [../../ncnn/src/layer/multiheadattention.h](../../ncnn/src/layer/multiheadattention.h) / [multiheadattention.cpp](../../ncnn/src/layer/multiheadattention.cpp) | MHA with cache, int8 and block-quant paths | +| [../../ncnn/src/layer/rotaryembed.h](../../ncnn/src/layer/rotaryembed.h) / [rotaryembed.cpp](../../ncnn/src/layer/rotaryembed.cpp) | RoPE from host cos/sin caches | +| [../../ncnn/src/layer/embed.h](../../ncnn/src/layer/embed.h) | Token embedding lookup | +| [../../ncnn/src/layer/gemm.h](../../ncnn/src/layer/gemm.h) | Gemm params incl. block quantization | +| [../../ncnn/src/layer/spectrogram.h](../../ncnn/src/layer/spectrogram.h) / [spectrogram.cpp](../../ncnn/src/layer/spectrogram.cpp) / [inversespectrogram.h](../../ncnn/src/layer/inversespectrogram.h) / [inversespectrogram.cpp](../../ncnn/src/layer/inversespectrogram.cpp) | STFT / iSTFT layers | +| [../../ncnn/src/layer/arm](../../ncnn/src/layer/arm) / [x86](../../ncnn/src/layer/x86) / [riscv](../../ncnn/src/layer/riscv) / [mips](../../ncnn/src/layer/mips) / [loongarch](../../ncnn/src/layer/loongarch) | Arch-optimized kernels | +| [../../ncnn/src/layer/arm/sdpa_arm.cpp](../../ncnn/src/layer/arm/sdpa_arm.cpp) | ARM SDPA (fp16/bf16 storage flags) | +| [../../ncnn/src/layer/arm/convolution_arm_i8mm.cpp](../../ncnn/src/layer/arm/convolution_arm_i8mm.cpp) / [convolution1d_arm_asimdhp.cpp](../../ncnn/src/layer/arm/convolution1d_arm_asimdhp.cpp) | ISA-variant kernel examples | +| [../../ncnn/src/layer/x86/convolution_x86_avx512vnni.cpp](../../ncnn/src/layer/x86/convolution_x86_avx512vnni.cpp) | x86 int8 VNNI kernel | +| [../../ncnn/src/layer/vulkan](../../ncnn/src/layer/vulkan) / [shader](../../ncnn/src/layer/vulkan/shader) | Vulkan layer impls and 300 GLSL shaders | +| [../../ncnn/src/layer/vulkan/shader/sdpa_kvcache_append.comp](../../ncnn/src/layer/vulkan/shader/sdpa_kvcache_append.comp) | GPU kv-cache append shader | +| [../../ncnn/src/layer/vulkan/convolution1d_vulkan.cpp](../../ncnn/src/layer/vulkan/convolution1d_vulkan.cpp) | 1-D conv on GPU | +| [../../ncnn/cmake/ncnn_add_layer.cmake](../../ncnn/cmake/ncnn_add_layer.cmake) / [ncnn_add_shader.cmake](../../ncnn/cmake/ncnn_add_shader.cmake) / [run_test.cmake](../../ncnn/cmake/run_test.cmake) | Build macros | +| [../../ncnn/toolchains](../../ncnn/toolchains) / [jetson.toolchain.cmake](../../ncnn/toolchains/jetson.toolchain.cmake) / [ios.toolchain.cmake](../../ncnn/toolchains/ios.toolchain.cmake) | Cross-compile toolchains | +| [../../ncnn/tools/pnnx](../../ncnn/tools/pnnx) / [README.md](../../ncnn/tools/pnnx/README.md) / [src/save_ncnn.cpp](../../ncnn/tools/pnnx/src/save_ncnn.cpp) / [src/load_onnx.cpp](../../ncnn/tools/pnnx/src/load_onnx.cpp) / [src/load_tnn.cpp](../../ncnn/tools/pnnx/src/load_tnn.cpp) / [src/pass_ncnn](../../ncnn/tools/pnnx/src/pass_ncnn) | PyTorch/ONNX → ncnn exporter | +| [../../ncnn/tools/pnnx/src/pass_ncnn/torch_stft.cpp](../../ncnn/tools/pnnx/src/pass_ncnn/torch_stft.cpp) / [torchaudio_F_inverse_spectrogram.cpp](../../ncnn/tools/pnnx/src/pass_ncnn/torchaudio_F_inverse_spectrogram.cpp) | Audio op lowering | +| [../../ncnn/tools/onnx/onnx2ncnn.cpp](../../ncnn/tools/onnx/onnx2ncnn.cpp) | Legacy ONNX converter | +| [../../ncnn/tools/caffe](../../ncnn/tools/caffe) / [darknet](../../ncnn/tools/darknet) / [mxnet](../../ncnn/tools/mxnet) / [keras](../../ncnn/tools/keras) / [tensorflow](../../ncnn/tools/tensorflow) / [mlir](../../ncnn/tools/mlir) | Other converters | +| [../../ncnn/tools/ncnnoptimize.cpp](../../ncnn/tools/ncnnoptimize.cpp) / [ncnnmerge.cpp](../../ncnn/tools/ncnnmerge.cpp) / [ncnn2mem.cpp](../../ncnn/tools/ncnn2mem.cpp) | Offline graph optimization / packaging | +| [../../ncnn/tools/quantize](../../ncnn/tools/quantize) / [ncnnllm2table.cpp](../../ncnn/tools/quantize/ncnnllm2table.cpp) / [ncnnllm2int.cpp](../../ncnn/tools/quantize/ncnnllm2int.cpp) | PTQ int8 and LLM block quantization tools | +| [../../ncnn/benchmark/benchncnn.cpp](../../ncnn/benchmark/benchncnn.cpp) / [benchncnn_llm.cpp](../../ncnn/benchmark/benchncnn_llm.cpp) / [CMakeLists.txt](../../ncnn/benchmark/CMakeLists.txt) / [README.md](../../ncnn/benchmark/README.md) / [models/vision_transformer.param](../../ncnn/benchmark/models/vision_transformer.param) | Benchmarks and repo-reported results | +| [../../ncnn/examples/CMakeLists.txt](../../ncnn/examples/CMakeLists.txt) | Example targets | +| [../../ncnn/examples/piper.cpp](../../ncnn/examples/piper.cpp) | Piper/VITS TTS example | +| [../../ncnn/examples/whisper.cpp](../../ncnn/examples/whisper.cpp) | Whisper ASR with kv cache and beam search | +| [../../ncnn/examples/yolo11.cpp](../../ncnn/examples/yolo11.cpp) / [rvm.cpp](../../ncnn/examples/rvm.cpp) / [squeezencnn/README.md](../../ncnn/examples/squeezencnn/README.md) | Vision / recurrent-state / Android pointers | +| [../../ncnn/tests/CMakeLists.txt](../../ncnn/tests/CMakeLists.txt) / [testutil.h](../../ncnn/tests/testutil.h) | Test harness | +| [../../ncnn/tests/test_sdpa_kvcache.cpp](../../ncnn/tests/test_sdpa_kvcache.cpp) / [test_sdpa_kvcache_session.cpp](../../ncnn/tests/test_sdpa_kvcache_session.cpp) / [test_multiheadattention_kvcache.cpp](../../ncnn/tests/test_multiheadattention_kvcache.cpp) / [test_multiheadattention_kvcache_allocator.cpp](../../ncnn/tests/test_multiheadattention_kvcache_allocator.cpp) / [test_gemm_block_quant.cpp](../../ncnn/tests/test_gemm_block_quant.cpp) / [test_mat_batch.cpp](../../ncnn/tests/test_mat_batch.cpp) | Transformer/kv-cache/batch tests | +| [../../ncnn/tests/perf](../../ncnn/tests/perf) / [perf_sdpa_kvcache.cpp](../../ncnn/tests/perf/perf_sdpa_kvcache.cpp) / [perf_sdpa_decode.cpp](../../ncnn/tests/perf/perf_sdpa_decode.cpp) / [perf_sdpa_prefill.cpp](../../ncnn/tests/perf/perf_sdpa_prefill.cpp) | Micro-benchmarks | +| [../../ncnn/python/src/main.cpp](../../ncnn/python/src/main.cpp) / [python/README.md](../../ncnn/python/README.md) / [python/ncnn/model_zoo](../../ncnn/python/ncnn/model_zoo) / [python/examples](../../ncnn/python/examples) | Python bindings | +| [../../ncnn/.github/workflows](../../ncnn/.github/workflows) / [compare-binary-size.yml](../../ncnn/.github/workflows/compare-binary-size.yml) | CI matrix, binary-size tracking | +| [../../ncnn/docs/developer-guide/kvcache.md](../../ncnn/docs/developer-guide/kvcache.md) | KV cache design and usage | +| [../../ncnn/docs/developer-guide/layer-support-behavior.md](../../ncnn/docs/developer-guide/layer-support-behavior.md) | Meaning of `support_*` flags | +| [../../ncnn/docs/developer-guide/layer-feat-mask.md](../../ncnn/docs/developer-guide/layer-feat-mask.md) | Per-layer feature mask `31=` | +| [../../ncnn/docs/developer-guide/expression.md](../../ncnn/docs/developer-guide/expression.md) | Dynamic shape expressions | +| [../../ncnn/docs/developer-guide/element-packing.md](../../ncnn/docs/developer-guide/element-packing.md) | elempack layout | +| [../../ncnn/docs/developer-guide/param-and-model-file-structure.md](../../ncnn/docs/developer-guide/param-and-model-file-structure.md) | File formats | +| [../../ncnn/docs/developer-guide/operators.md](../../ncnn/docs/developer-guide/operators.md) / [operation-param-weight-table.md](../../ncnn/docs/developer-guide/operation-param-weight-table.md) | Operator/param reference | +| [../../ncnn/docs/developer-guide/custom-allocator.md](../../ncnn/docs/developer-guide/custom-allocator.md) | Allocator roles and concurrency patterns | +| [../../ncnn/docs/developer-guide/low-level-operation-api.md](../../ncnn/docs/developer-guide/low-level-operation-api.md) | Single-layer usage | +| [../../ncnn/docs/developer-guide/how-to-implement-custom-layer-step-by-step.md](../../ncnn/docs/developer-guide/how-to-implement-custom-layer-step-by-step.md) / [add-custom-layer.zh.md](../../ncnn/docs/developer-guide/add-custom-layer.zh.md) | Custom layer tutorials | +| [../../ncnn/docs/developer-guide/vulkan-driver-loader.md](../../ncnn/docs/developer-guide/vulkan-driver-loader.md) / [vulkan-pipeline-cache.md](../../ncnn/docs/developer-guide/vulkan-pipeline-cache.md) / [glsl-extension.md](../../ncnn/docs/developer-guide/glsl-extension.md) | Vulkan internals | +| [../../ncnn/docs/developer-guide/arm-a53-a55-dual-issue.md](../../ncnn/docs/developer-guide/arm-a53-a55-dual-issue.md) | Little-core kernel tuning | +| [../../ncnn/docs/how-to-use-and-FAQ/quantized-int8-inference.md](../../ncnn/docs/how-to-use-and-FAQ/quantized-int8-inference.md) | PTQ and LLM block quant workflow | +| [../../ncnn/docs/how-to-use-and-FAQ/build-minimal-library.md](../../ncnn/docs/how-to-use-and-FAQ/build-minimal-library.md) | Binary size reduction | +| [../../ncnn/docs/how-to-use-and-FAQ/openmp-best-practice.md](../../ncnn/docs/how-to-use-and-FAQ/openmp-best-practice.md) | Threading/affinity guidance | +| [../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md) / [FAQ-ncnn-vulkan.md](../../ncnn/docs/how-to-use-and-FAQ/FAQ-ncnn-vulkan.md) | Vulkan usage, platform matrix, fallback | +| [../../ncnn/docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md](../../ncnn/docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md) | Zero-copy camera input | +| [../../ncnn/docs/how-to-use-and-FAQ/ncnn-load-model.md](../../ncnn/docs/how-to-use-and-FAQ/ncnn-load-model.md) / [efficient-roi-resize-rotate.md](../../ncnn/docs/how-to-use-and-FAQ/efficient-roi-resize-rotate.md) | Loading and preprocessing docs | +| [../../ncnn/docs/how-to-build/how-to-build.md](../../ncnn/docs/how-to-build/how-to-build.md) | Per-platform build instructions (Jetson, Android, iOS, WASM, HarmonyOS, ESP32…) | diff --git a/docs/engines/repository_inventory.md b/docs/engines/repository_inventory.md new file mode 100644 index 000000000..0bf557c5a --- /dev/null +++ b/docs/engines/repository_inventory.md @@ -0,0 +1,50 @@ +# Repository inventory + +Verified on 2026-09-08 (UTC+8 machine clock) with `git rev-parse`, `git status --porcelain`, `git remote -v`, and `git submodule status`. No checkout was modified. The analysis root is `/data/zhoutaichang/embedding_infer/`; `/home/zhoutaichang/embedding_infer` is the same directory (same inode), so both paths refer to one set of checkouts. + +## Expected vs actual + +| Engine | Local path | Remote | Expected branch / HEAD | Actual branch | Actual HEAD | Working tree | Last commit (author date) | +|---|---|---|---|---|---|---|---| +| vLLM | [`../vllm`](../vllm) | `https://github.com/vllm-project/vllm.git` | main / 51da0ca66 | `main` | `51da0ca66c8065619c79e35dff97aa99aeaf5644` | clean | 2026-09-07 15:15 UTC, "[CI/Build] Unskip ColQwen3 multimodal pooling tests on Transformers v5 (#55588)" | +| vLLM-Omni | [`../vllm-omni`](../vllm-omni) | `https://github.com/vllm-project/vllm-omni.git` | main / 7be014bc | `main` | `7be014bce6374f06c95b703763bdbac4c6198f31` | clean | 2026-09-07 14:50 UTC, "[Bugfix][NPU] Fix MiniMax-H3 INT8 quantization dispatch (#6876)" | +| llama.cpp | [`../llama.cpp`](../llama.cpp) | `https://github.com/ggml-org/llama.cpp.git` | master / e71b80510 | `master` | `e71b80510c848c00175924ecf3c40333ccae8eb5` | clean | 2026-09-07 16:28 +0200, Revert "CUDA: size routed MoE MMQ N-tiles ... (#24546)" (#28551) | +| ncnn | [`../ncnn`](../ncnn) | `https://github.com/Tencent/ncnn.git` | master / 3b7bdba7 | `master` | `3b7bdba7fc8aea8fd46779533eee027df77c639d` | clean | 2026-09-07 19:42 +0800, "Fix SIGFPE with missing or incomplete CPU topology (#6967)" | +| MNN | [`../MNN`](../MNN) | `https://github.com/alibaba/MNN.git` | master / bef71b97 | `master` | `bef71b9756a2c77549eddbe33eb97290e3b16602` | clean | 2026-09-04 17:39 +0800, "[Vulkan:Perf] Optimize INT4 cooperative matrix path" | +| ExecuTorch | [`../executorch`](../executorch) | `https://github.com/pytorch/executorch.git` | main / cfc6ecd516 | `main` | `cfc6ecd5166bec70c5fde2bad836fed26018fe52` | clean | 2026-09-07 08:47 -0700, "Qualcomm AI Engine Direct - Enable QNN Windows ARM64 build in CI (#22438)" | + +**Result: all six checkouts match the expected branch and commit exactly, and all six working trees are clean.** No discrepancies in branch or HEAD. + +## Discrepancies and caveats (not corrected, reported only) + +1. **Submodules are not initialized** in the checkouts that declare them (`git submodule status` prints a leading `-`): + - ncnn: `glslang`, `python/pybind11`. Consequence: a Vulkan-enabled build of ncnn from this tree would need `git submodule update --init` (not run). CPU-only analysis is unaffected. + - executorch: 21 submodules including `backends/xnnpack/third-party/XNNPACK`, `backends/vulkan/third-party/*`, `backends/mlx/third-party/mlx`, `extension/llm/tokenizers`, `third-party/{flatbuffers,flatcc,googletest,gflags,json,ao,ios-cmake}`, `kernels/optimized/third-party/eigen`, `shim`, `backends/cadence/utils/FACTO`. Consequence: the tokenizer implementation used by ExecuTorch's LLM runner (`extension/llm/tokenizers`) is **not present on disk**, so statements about it in [engines/executorch.md](engines/executorch.md) are limited to how it is called, not how it is implemented. + - vLLM, vLLM-Omni, llama.cpp, MNN declare no submodules at these commits (empty `git submodule status`). +2. **Installed Python packages differ from the analyzed checkouts.** The machine's Python 3.12 has `vllm 0.28.0` installed as a wheel in `~/.local/lib/python3.12/site-packages`, and `vllm-omni 0.28.0rc2.dev22+gbe335a86f` installed editable from `/home/zhoutaichang/feature/vllm-omni-main` (commit `be335a86f`), **not** from `../vllm-omni` (`7be014bc`). Any runtime experiment therefore exercises the installed versions, not the analyzed commits, unless a separate environment is created (which would be a dependency installation and was not done without permission). See [experiments.md](experiments.md). +3. **Host is x86_64, not ARM.** Native ARM CPU, Apple Silicon, and mobile GPU/NPU measurements cannot be taken on this machine; only x86 CPU and NVIDIA L20X data-center GPU runs are possible here. Edge numbers in this study are therefore either repo-reported (cited) or marked as unknown. + +## Tree size snapshot + +Rough sizes to calibrate reading effort (from `du -sh`, excluding `.git`); see per-engine reports for module maps. + +| Engine | Primary language(s) | Notable top-level dirs | +|---|---|---| +| vLLM | Python, CUDA/C++ (`csrc/`) | `vllm/v1`, `vllm/model_executor`, `vllm/platforms`, `vllm/multimodal`, `vllm/entrypoints`, `csrc/`, `docs/` | +| vLLM-Omni | Python | `vllm_omni/{engine,core,worker,diffusion,model_executor,entrypoints,distributed,config,host_weight_runtime}`, `examples/`, `docs/`, `recipes/` | +| llama.cpp | C/C++ | `ggml/`, `src/`, `include/`, `tools/` (server, mtmd, tts), `examples/`, `gguf-py/`, `convert_hf_to_gguf.py` | +| ncnn | C++ (+GLSL shaders) | `src/` (`layer/{arm,x86,vulkan,...}`), `tools/` (pnnx, quantize), `python/`, `examples/`, `docs/` | +| MNN | C++ (+Metal/OpenCL/Vulkan/CUDA), Python | `source/{core,backend,geometry,shape}`, `express/`, `schema/default`, `tools/converter`, `transformers/{llm,diffusion}`, `apps/`, `pymnn/` | +| ExecuTorch | Python (AOT), C++ (runtime) | `exir/`, `runtime/`, `backends/`, `kernels/`, `extension/`, `examples/`, `devtools/`, `docs/source` | + +## Per-repo agent guidance honored during analysis + +- `vllm/AGENTS.md` (via `CLAUDE.md`): no system `pip`; use `uv`. No packages were installed for this study. +- `llama.cpp/AGENTS.md`: fully autonomous agents must not contribute; this study is read-only analysis and proposes no PRs. +- `MNN/AGENTS.md`: `schema/private/` and `source/internal/` are restricted and were not read or referenced. +- `executorch/AGENTS.md`: points to `.claude/` reference docs and `.wiki/`; used as documentation sources. + +## Related material outside the six checkouts + +- GPU scheduling guide: `/home/zhoutaichang/feature/GPU调度工具快速上手.pdf` (read; procedures summarized in [experiments.md](experiments.md)). +- Other vLLM-Omni worktrees exist under `/home/zhoutaichang/feature/` (`vllm-omni-main`, `vllm-omni-abot`, `vllm-omni-pr6463`, ...). They were **not** analyzed; only `../vllm-omni` at `7be014bc` is in scope. diff --git a/docs/engines/vllm.md b/docs/engines/vllm.md new file mode 100644 index 000000000..ebc3abfe4 --- /dev/null +++ b/docs/engines/vllm.md @@ -0,0 +1,484 @@ +# vLLM (vllm-project/vllm) — Engine Analysis + +| Field | Value | +| --- | --- | +| Local path | `/data/zhoutaichang/embedding_infer/vllm` | +| Remote | `https://github.com/vllm-project/vllm.git` (origin) | +| Branch | `main` | +| HEAD | `51da0ca66c8065619c79e35dff97aa99aeaf5644` — 2026-09-07 "[CI/Build] Unskip ColQwen3 multimodal pooling tests on Transformers v5 (#55588)" | +| `git describe --tags` | `v0.28.1rc0-500-g51da0ca66` (newest tag in checkout: `v0.29.0rc4`) | +| Working tree | clean (`git status --porcelain` empty) | +| Submodules | none registered (`git submodule status` prints nothing); kernel deps are fetched by CMake `FetchContent` at build time | +| Installed vs checkout | `pip show vllm` reports **0.28.0** installed in this machine's Python, while the checkout is ~500 commits past `v0.28.1rc0` (post-0.29.0rc4). Importing `vllm` from inside the checkout fails (`No module named 'vllm._C_stable_libtorch'`) because the C++/CUDA extensions are not built here. All statements below are about the **checkout**, not the installed wheel. | +| Analysis date | 2026-09-08 | +| Method | Read-only source/doc inspection (no builds, no execution). Evidence tags: [Verified] / [Inferred] / [Proposal] / [Unknown]. | + +Reading guidance followed: `AGENTS.md` (`CLAUDE.md` is just `@AGENTS.md`) — no private/internal dirs are excluded for this repo. + +## 0. Summary + +- [Verified] vLLM is a Python-first LLM **serving stack** (OpenAI-compatible HTTP/gRPC/Python API) plus an **execution runtime** built on PyTorch: request scheduling + paged KV cache in an "EngineCore" process, model execution in worker processes, and hand-written CUDA/C++/Triton kernels bound through `torch.ops`. There is no standalone C/C++ runtime binary; PyTorch (`torch==2.13.0`) is a hard dependency ([pyproject.toml](../../vllm/pyproject.toml), [requirements/common.txt](../../vllm/requirements/common.txt)). +- [Verified] Only the **V1 engine** exists in-tree: `AsyncLLM`/`LLMEngine` → `EngineCoreClient` (ZMQ + msgspec) → `EngineCore` (`Scheduler` + `KVCacheManager`) → `Executor` → `Worker` → `GPUModelRunner` ([vllm/v1/engine/async_llm.py](../../vllm/vllm/v1/engine/async_llm.py), [vllm/v1/engine/core.py](../../vllm/vllm/v1/engine/core.py)). +- [Verified] In-tree platforms: **CUDA, ROCm, XPU (Intel GPU), CPU (x86/ARM/PowerPC/s390x/RISC-V, macOS CPU)** and a **Zen-CPU** variant. **TPU is a thin shim** that imports the out-of-tree `tpu_inference` package. Everything else (Ascend NPU, Apple Metal/MLX, Gaudi, ...) is an out-of-tree platform plugin discovered via the `vllm.platform_plugins` entry-point group ([vllm/platforms/__init__.py](../../vllm/vllm/platforms/__init__.py), [vllm/platforms/tpu.py](../../vllm/vllm/platforms/tpu.py)). +- [Verified] Backend substitution has three documented extension points: platform plugin (`Platform` subclass returning a `worker_cls`, attention backend, device communicator), out-of-tree models (`ModelRegistry.register_model` via `vllm.general_plugins`), and out-of-tree layer/op implementations (`CustomOp.register_oot`, `AttentionBackendEnum.CUSTOM`, and the newer `vllm.ir` op/impl registry) ([docs/design/plugin_system.md](../../vllm/docs/design/plugin_system.md), [vllm/model_executor/custom_op.py](../../vllm/vllm/model_executor/custom_op.py), [vllm/ir/op.py](../../vllm/vllm/ir/op.py)). +- [Verified] Request scheduling is a single token-budget loop with **no explicit prefill/decode phases** (chunked prefill, prefix caching, spec decode all fall out of `num_computed_tokens` vs `num_tokens_with_spec`), FCFS or priority queues, preemption-by-recompute, and an `AsyncScheduler` subclass that overlaps step N+1 scheduling with step N execution ([vllm/v1/core/sched/scheduler.py](../../vllm/vllm/v1/core/sched/scheduler.py#L535)). +- [Verified] KV cache is block-paged with hash-based prefix caching (sha256 default), LRU eviction via a doubly linked free list, and per-layer-type managers (full, sliding-window, chunked-local, Mamba, circular buffer) coordinated by a hybrid coordinator ([vllm/v1/core/block_pool.py](../../vllm/vllm/v1/core/block_pool.py), [vllm/v1/core/single_type_kv_cache_manager.py](../../vllm/vllm/v1/core/single_type_kv_cache_manager.py)). +- [Verified] Execution uses `torch.compile` (piecewise Dynamo/Inductor, "VLLM_COMPILE" mode) plus a CUDA-graph dispatcher with `NONE/PIECEWISE/FULL/FULL_DECODE_ONLY/FULL_AND_PIECEWISE` modes; all compilation/capture happens at startup, never on a live request ([vllm/config/compilation.py](../../vllm/vllm/config/compilation.py), [vllm/v1/cudagraph_dispatcher.py](../../vllm/vllm/v1/cudagraph_dispatcher.py), [docs/design/cuda_graphs.md](../../vllm/docs/design/cuda_graphs.md)). +- [Verified] A second, experimental **Model Runner V2** (`vllm/v1/worker/gpu/`) is now the default when Triton is available and no unsupported feature is requested; it is "async-first" and Triton-heavy ([vllm/config/vllm.py](../../vllm/vllm/config/vllm.py#L657), [docs/design/model_runner_v2.md](../../vllm/docs/design/model_runner_v2.md)). +- [Verified] **CPU backend** is real and ARM-aware: NEON/BF16/I8MM code paths in `csrc/cpu`, oneDNN + Arm Compute Library GEMMs, KleidiAI-backed dynamic-4-bit W4A8 kernels through `torch.ops.aten._dyn_quant_matmul_4bit`, thread binding, tcmalloc preload, aarch64 wheels and `arm64` Docker images; validated on AWS Graviton3 per docs ([cmake/cpu_extension.cmake](../../vllm/cmake/cpu_extension.cmake), [vllm/v1/attention/backends/cpu_attn.py](../../vllm/vllm/v1/attention/backends/cpu_attn.py#L481), [docs/getting_started/installation/cpu.arm.inc.md](../../vllm/docs/getting_started/installation/cpu.arm.inc.md)). +- [Verified] **TTS: none.** There is no speech-synthesis model, vocoder, codec decoder, or audio-output endpoint. What exists is audio **input** (Whisper, Ultravox, Qwen2-Audio, Voxtral, Kimi-Audio, MOSS-Audio, omni "thinker" halves, etc.), `/v1/audio/transcriptions|translations`, and a WebSocket `/v1/realtime` streaming-ASR path with a `SupportsRealtime` model interface ([vllm/model_executor/models/registry.py](../../vllm/vllm/model_executor/models/registry.py), [vllm/entrypoints/speech_to_text/realtime/serving.py](../../vllm/vllm/entrypoints/speech_to_text/realtime/serving.py)). +- [Verified] **VLA: one model.** `OpenVLAForActionPrediction` (DINOv2+SigLIP backbone, projector, Llama LM, discretized action-token vocabulary) is registered; no proprioceptive input channel, no diffusion/flow action head, no closed-loop control API. `DiffusionGemmaForBlockDiffusion` shows the runner can host a non-autoregressive block-diffusion *text* decoder ([vllm/model_executor/models/openvla.py](../../vllm/vllm/model_executor/models/openvla.py), [vllm/model_executor/models/diffusion_gemma.py](../../vllm/vllm/model_executor/models/diffusion_gemma.py)). +- [Verified] Jetson is recognized (`/etc/nv_tegra_release` check, non-NVML CUDA platform, `8.7`/`11.0` in supported CUDA archs, Thor `sm_110a` troubleshooting note) but there is no Jetson-specific build target, docs page, or CI job ([vllm/platforms/__init__.py](../../vllm/vllm/platforms/__init__.py#L76), [CMakeLists.txt](../../vllm/CMakeLists.txt#L112), [docs/usage/troubleshooting.md](../../vllm/docs/usage/troubleshooting.md)). +- [Verified] Apple Silicon: CPU-only in-tree (FP16/FP32, BF16 if `FEAT_BF16`, build from source, CI smoke-tested on macOS runners); GPU via the out-of-tree `vllm-metal` (MLX) plugin. No Android/Vulkan/OpenCL/QNN/NNAPI code anywhere in tree ([docs/getting_started/installation/cpu.apple.inc.md](../../vllm/docs/getting_started/installation/cpu.apple.inc.md), [docs/getting_started/installation/gpu.apple.inc.md](../../vllm/docs/getting_started/installation/gpu.apple.inc.md)). +- [Verified] GGUF and bitsandbytes quantization have been **moved out of tree** to plugins (`vllm-gguf-plugin`, `vllm-bnb-plugin`); in-tree formats are AWQ/GPTQ(+Marlin), FP8, compressed-tensors (INT8 W8A8, W4A8 int/fp8, ...), ModelOpt (FP4/MXFP8), MXFP4, TorchAO, Quark, INC, and an "online" method ([vllm/model_executor/layers/quantization/__init__.py](../../vllm/vllm/model_executor/layers/quantization/__init__.py), [docs/features/quantization/gguf.md](../../vllm/docs/features/quantization/gguf.md)). +- [Verified] Deployment footprint is server-class: 3+ processes (API server, engine core, one worker per device), ZMQ/msgspec IPC, FastAPI/uvicorn, `transformers>=5.10`, `torch 2.13`; wheel size CI cap is 500 MB ([docs/design/arch_overview.md](../../vllm/docs/design/arch_overview.md#L67), [.buildkite/check-wheel-size.py](../../vllm/.buildkite/check-wheel-size.py)). + +## 1. Architecture overview + +```mermaid +graph TD + subgraph FE["Frontend process(es)"] + CLI["vllm CLI
vllm/entrypoints/cli"] --> API["OpenAI-compatible FastAPI server
vllm/entrypoints/launchers/api_server
routers: chat/completions/responses/embeddings/
audio transcriptions/translations/realtime WS"] + PY["LLM (offline)
vllm/entrypoints/llm.py"] --> LLME["LLMEngine (sync)
vllm/v1/engine/llm_engine.py"] + RS["Rust frontend (experimental)
rust/"] -.ZMQ.-> CORE + API --> ALLM["AsyncLLM
vllm/v1/engine/async_llm.py"] + ALLM --> IP["InputProcessor + Renderer
tokenize, MM preprocess, validate"] + ALLM --> OP["OutputProcessor + IncrementalDetokenizer"] + IP --> CC["EngineCoreClient (Inproc / Sync MP / Async MP)
vllm/v1/engine/core_client.py"] + LLME --> CC + end + CC -- "ZMQ + msgspec" --> CORE + subgraph EC["EngineCore process (1 per DP rank)"] + CORE["EngineCoreProc.run_busy_loop
vllm/v1/engine/core.py"] --> SCH["Scheduler / AsyncScheduler
vllm/v1/core/sched/"] + SCH --> KVM["KVCacheManager → KVCacheCoordinator → BlockPool
prefix cache, LRU free list"] + SCH --> ECM["EncoderCacheManager"] + SCH --> SOM["StructuredOutputManager
xgrammar/guidance/outlines/lmfe"] + CORE --> EXE["Executor (Uniproc / Multiproc / Ray)
vllm/v1/executor/"] + end + EXE -- "shm MessageQueue / RPC" --> W + subgraph WK["Worker process (1 per device)"] + W["Worker (gpu_worker / cpu_worker / xpu_worker)"] --> MR["GPUModelRunner V1 (gpu_model_runner.py)
or Model Runner V2 (vllm/v1/worker/gpu/)"] + MR --> CG["CudagraphDispatcher + torch.compile backend
vllm/v1/cudagraph_dispatcher.py, vllm/compilation/"] + MR --> MODEL["nn.Module from ModelRegistry
vllm/model_executor/models/"] + MODEL --> LAYERS["Layers: Attention, Linear, MoE, RMSNorm, RoPE, quant
vllm/model_executor/layers/"] + LAYERS --> ATTN["AttentionBackend (FA/FlashInfer/Triton/FlexAttention/
ROCm/CPU/...) via AttentionBackendEnum"] + LAYERS --> OPS["CustomOp dispatch → forward_cuda/cpu/xpu/hip/oot
vllm.ir ops → priority impls (vllm_c/native/...)"] + OPS --> CSRC["torch.ops._C / _C_stable_libtorch / _moe_C
csrc/, csrc/cpu/, csrc/libtorch_stable/"] + MR --> SMP["Sampler (vllm/v1/sample/)"] + W --> KVT["KV cache tensors; KVConnector / KV offload"] + end + PLAT["current_platform (vllm/platforms)
builtin: cuda, rocm, xpu, cpu, zen_cpu, tpu(shim)
entry-point group vllm.platform_plugins"] -. "worker_cls, attn backend,
communicator, dtype/quant checks" .-> W + PLAT -.-> ATTN +``` + +Key modules: + +| Module | Path | Responsibility | Key symbols | +| --- | --- | --- | --- | +| Async engine front-end | [vllm/v1/engine/async_llm.py](../../vllm/vllm/v1/engine/async_llm.py) | Request admission, streaming outputs, abort/pause, streaming-input sessions | `AsyncLLM`, `generate`, `_add_streaming_input_request`, `abort` | +| Sync engine | [vllm/v1/engine/llm_engine.py](../../vllm/vllm/v1/engine/llm_engine.py) | Offline `step()` loop used by `LLM` | `LLMEngine.step`, `add_request` | +| Engine core | [vllm/v1/engine/core.py](../../vllm/vllm/v1/engine/core.py) | Owns scheduler, KV init, busy loop, ZMQ I/O threads | `EngineCore.step`, `step_with_batch_queue`, `_initialize_kv_caches`, `EngineCoreProc.run_busy_loop` | +| Core client | [vllm/v1/engine/core_client.py](../../vllm/vllm/v1/engine/core_client.py) | In-proc vs multiproc clients, request/response transport | `EngineCoreClient.make_client`, `InprocClient` | +| Input/output processing | [vllm/v1/engine/input_processor.py](../../vllm/vllm/v1/engine/input_processor.py), [vllm/v1/engine/output_processor.py](../../vllm/vllm/v1/engine/output_processor.py), [vllm/v1/engine/detokenizer.py](../../vllm/vllm/v1/engine/detokenizer.py) | Validation, MM cache injection, incremental detokenization, stop strings | `InputProcessor.process_inputs`, `OutputProcessor.process_outputs`, `FastIncrementalDetokenizer` | +| Scheduler | [vllm/v1/core/sched/scheduler.py](../../vllm/vllm/v1/core/sched/scheduler.py), [vllm/v1/core/sched/async_scheduler.py](../../vllm/vllm/v1/core/sched/async_scheduler.py), [vllm/v1/core/sched/request_queue.py](../../vllm/vllm/v1/core/sched/request_queue.py) | Token-budget scheduling, preemption, encoder budget, spec-decode slots | `Scheduler.schedule`, `update_from_output`, `_preempt_request`, `FCFSRequestQueue`, `PriorityRequestQueue` | +| KV cache manager | [vllm/v1/core/kv_cache_manager.py](../../vllm/vllm/v1/core/kv_cache_manager.py), [vllm/v1/core/kv_cache_coordinator.py](../../vllm/vllm/v1/core/kv_cache_coordinator.py), [vllm/v1/core/block_pool.py](../../vllm/vllm/v1/core/block_pool.py), [vllm/v1/core/kv_cache_utils.py](../../vllm/vllm/v1/core/kv_cache_utils.py) | Block allocation, prefix-cache lookup, eviction, events | `KVCacheManager.allocate_slots`, `get_computed_blocks`, `BlockPool.cache_full_blocks`, `FreeKVCacheBlockQueue`, `hash_block_tokens` | +| KV cache specs | [vllm/v1/kv_cache_interface.py](../../vllm/vllm/v1/kv_cache_interface.py), [vllm/v1/kv_cache_layout.py](../../vllm/vllm/v1/kv_cache_layout.py) | Per-layer spec types, physical layouts | `FullAttentionSpec`, `SlidingWindowSpec`, `MambaSpec`, `KVCacheConfig`, `KVCacheLayout` | +| Executor | [vllm/v1/executor/abstract.py](../../vllm/vllm/v1/executor/abstract.py), [vllm/v1/executor/multiproc_executor.py](../../vllm/vllm/v1/executor/multiproc_executor.py), [vllm/v1/executor/uniproc_executor.py](../../vllm/vllm/v1/executor/uniproc_executor.py) | Spawns workers, broadcasts `execute_model`/`sample_tokens` | `Executor.get_class`, `MultiprocExecutor`, `WorkerProc` | +| Workers | [vllm/v1/worker/worker_base.py](../../vllm/vllm/v1/worker/worker_base.py), [vllm/v1/worker/gpu_worker.py](../../vllm/vllm/v1/worker/gpu_worker.py), [vllm/v1/worker/cpu_worker.py](../../vllm/vllm/v1/worker/cpu_worker.py), [vllm/v1/worker/xpu_worker.py](../../vllm/vllm/v1/worker/xpu_worker.py) | Device init, memory profiling, warmup/graph capture, sleep mode | `Worker.init_device`, `determine_available_memory`, `compile_or_warm_up_model`, `CPUWorker` | +| Model runner V1 | [vllm/v1/worker/gpu_model_runner.py](../../vllm/vllm/v1/worker/gpu_model_runner.py), [vllm/v1/worker/cpu_model_runner.py](../../vllm/vllm/v1/worker/cpu_model_runner.py) | Persistent batch, input prep, MM encoder execution, forward, sampling, KV alloc | `GPUModelRunner.execute_model`, `_dummy_run`, `capture_model`, `initialize_kv_cache`, `load_model`, `CPUModelRunner` | +| Model runner V2 | [vllm/v1/worker/gpu/model_runner.py](../../vllm/vllm/v1/worker/gpu/model_runner.py), [vllm/v1/worker/gpu/README.md](../../vllm/vllm/v1/worker/gpu/README.md) | Async-first rewrite; fixed-row persistent state, Gumbel sampling | `GPUModelRunner.execute_model`, `prepare_inputs`, `sample_tokens` | +| Attention | [vllm/v1/attention/backend.py](../../vllm/vllm/v1/attention/backend.py), [vllm/v1/attention/backends/registry.py](../../vllm/vllm/v1/attention/backends/registry.py), [vllm/v1/attention/selector.py](../../vllm/vllm/v1/attention/selector.py) | Backend ABC, enum registry with overrides, selection | `AttentionBackend`, `AttentionMetadataBuilder`, `AttentionBackendEnum`, `get_attn_backend` | +| Platforms | [vllm/platforms/interface.py](../../vllm/vllm/platforms/interface.py), [vllm/platforms/__init__.py](../../vllm/vllm/platforms/__init__.py), [vllm/platforms/cpu.py](../../vllm/vllm/platforms/cpu.py), [vllm/platforms/cuda.py](../../vllm/vllm/platforms/cuda.py) | Device abstraction and config mutation | `Platform`, `PlatformEnum`, `CpuArchEnum`, `resolve_current_platform_cls_qualname`, `CpuPlatform.check_and_update_config` | +| Custom ops / IR | [vllm/model_executor/custom_op.py](../../vllm/vllm/model_executor/custom_op.py), [vllm/ir/op.py](../../vllm/vllm/ir/op.py), [vllm/kernels/vllm_c.py](../../vllm/vllm/kernels/vllm_c.py) | Per-platform forward dispatch; provider-priority op impls | `CustomOp.dispatch_forward`, `register_oot`, `IrOp.register_impl`, `IrOp.dispatch` | +| Compilation | [vllm/compilation/backends.py](../../vllm/vllm/compilation/backends.py), [vllm/compilation/cuda_graph.py](../../vllm/vllm/compilation/cuda_graph.py), [vllm/compilation/decorators.py](../../vllm/vllm/compilation/decorators.py), [vllm/config/compilation.py](../../vllm/vllm/config/compilation.py) | Dynamo piecewise compile, Inductor passes, CUDA graph wrappers | `CompilationMode`, `CUDAGraphMode`, `CompilationConfig` | +| Quantization | [vllm/model_executor/layers/quantization/__init__.py](../../vllm/vllm/model_executor/layers/quantization/__init__.py), [vllm/model_executor/kernels/linear/mixed_precision/cpu.py](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py), [vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py) | Method registry, per-kernel `can_implement` selection | `QUANTIZATION_METHODS`, `CPUWNA16LinearKernel`, `Dynamic4bitLinearKernel` | +| Model registry | [vllm/model_executor/models/registry.py](../../vllm/vllm/model_executor/models/registry.py), [vllm/model_executor/models/interfaces.py](../../vllm/vllm/model_executor/models/interfaces.py) | Arch → module map, OOT registration, capability protocols | `_MULTIMODAL_MODELS`, `ModelRegistry.register_model`, `SupportsMultiModal`, `SupportsTranscription`, `SupportsRealtime` | +| Multimodal | [vllm/multimodal/registry.py](../../vllm/vllm/multimodal/registry.py), [vllm/multimodal/processing/processor.py](../../vllm/vllm/multimodal/processing/processor.py), [vllm/multimodal/cache.py](../../vllm/vllm/multimodal/cache.py), [vllm/multimodal/audio.py](../../vllm/vllm/multimodal/audio.py) | HF-processor wrapping, prompt updates, processor caches, audio resample/split | `MultiModalRegistry`, `BaseMultiModalProcessor`, `MultiModalProcessorCache*`, `AudioResampler`, `split_audio` | +| Sampling | [vllm/v1/sample/sampler.py](../../vllm/vllm/v1/sample/sampler.py), [vllm/v1/sample/ops/topk_topp_sampler.py](../../vllm/vllm/v1/sample/ops/topk_topp_sampler.py), [vllm/v1/sample/rejection_sampler.py](../../vllm/vllm/v1/sample/rejection_sampler.py) | Temperature/penalties/logits processors, top-k/p, spec-decode rejection | `Sampler.forward`, `TopKTopPSampler` | +| Entrypoints | [vllm/entrypoints/llm.py](../../vllm/vllm/entrypoints/llm.py), [vllm/entrypoints/launchers/api_server/entry.py](../../vllm/vllm/entrypoints/launchers/api_server/entry.py), [vllm/entrypoints/speech_to_text/base/serving.py](../../vllm/vllm/entrypoints/speech_to_text/base/serving.py), [vllm/entrypoints/grpc_server.py](../../vllm/vllm/entrypoints/grpc_server.py) | Python API, HTTP server, ASR endpoints, gRPC | `LLM.generate`, `run_server`, `SpeechToTextBaseServing`, `OpenAIServingRealtime` | +| Tokenizers | [vllm/tokenizers/registry.py](../../vllm/vllm/tokenizers/registry.py), [vllm/tokenizers/fastokens.py](../../vllm/vllm/tokenizers/fastokens.py) | HF / Mistral / custom tokenizer modes, optional Rust `fastokens` patch | `get_tokenizer`, `_TokenizerRegistry` | +| C++/CUDA kernels | [csrc/libtorch_stable/torch_bindings.cpp](../../vllm/csrc/libtorch_stable/torch_bindings.cpp), [csrc/cpu/torch_bindings.cpp](../../vllm/csrc/cpu/torch_bindings.cpp), [CMakeLists.txt](../../vllm/CMakeLists.txt), [cmake/cpu_extension.cmake](../../vllm/cmake/cpu_extension.cmake) | Op registration, build targets, ISA detection | `TORCH_LIBRARY_EXPAND`, `_C`, `_C_stable_libtorch`, `_moe_C_stable_libtorch` | + +## 2. Device layer + +**Platform abstraction.** [Verified] Everything device-specific is behind `vllm.platforms.current_platform`, an instance of a `Platform` subclass ([vllm/platforms/interface.py](../../vllm/vllm/platforms/interface.py#L134)). `PlatformEnum` has `CUDA, ROCM, TPU, XPU, CPU, OOT, UNSPECIFIED`; `CpuArchEnum` has `X86, ARM, POWERPC, S390X, RISCV, OTHER, UNKNOWN` ([interface.py#L67](../../vllm/vllm/platforms/interface.py#L67)). The interface (~1,340 lines) covers: `get_attn_backend_cls`, `get_vit_attn_backend`, `get_device_capability/name/uuid/total_memory`, `set_device`, `inference_mode`, `check_and_update_config` (mutates `VllmConfig`, sets `worker_cls`), `get_device_communicator_cls`, `get_punica_wrapper` (LoRA), `supports_fp8/mx`, `fp8_dtype`, `use_custom_allreduce`, `get_static_graph_wrapper_cls`, `support_static_graph_mode`, `import_kernels`, `import_ir_kernels`, `get_default_ir_op_priority`, `get_cpu_architecture`, `is_pin_memory_available`, `get_global_graph_pool`, `stateless_init_device_torch_dist_pg`. + +**Discovery/selection.** [Verified] `resolve_current_platform_cls_qualname()` ([vllm/platforms/__init__.py#L222](../../vllm/vllm/platforms/__init__.py#L222)) probes builtin detectors `{tpu, cuda, rocm, xpu, cpu}` and every `vllm.platform_plugins` entry point; exactly one may activate (two OOT or two builtin → `RuntimeError`), OOT wins over builtin, none → `UnspecifiedPlatform`. `VLLM_TARGET_DEVICE=cpu` short-circuits to CPU. Detection details: +- CUDA: `pynvml` device count > 0 and the wheel version string does not contain `cpu`; if NVML is missing, a **Jetson** check (`/etc/nv_tegra_release` or `/sys/class/tegra-firmware`) still enables CUDA ([__init__.py#L76](../../vllm/vllm/platforms/__init__.py#L76)). `CudaPlatform` is `NvmlCudaPlatform` or `NonNvmlCudaPlatform` chosen at import time ([cuda.py#L1040](../../vllm/vllm/platforms/cuda.py#L1040)). +- ROCm: `amdsmi` handles > 0, or WSL fallback via `torch.version.hip`. +- XPU: `torch.xpu.is_available()`. +- CPU: `VLLM_TARGET_DEVICE == "cpu"`, or a `+cpu` wheel version, or `sys.platform == darwin` (macOS is always CPU). AMD CPUs with AVX-512 and `zentorch` installed get `ZenCpuPlatform` ([vllm/platforms/zen_cpu.py](../../vllm/vllm/platforms/zen_cpu.py)). +- TPU: `libtpu` importable → `vllm.platforms.tpu.TpuPlatform`, which is just `from tpu_inference.platforms import TpuPlatform` ([vllm/platforms/tpu.py](../../vllm/vllm/platforms/tpu.py)); i.e. TPU execution code is **not** in this tree. + +**In-tree backend inventory.** [Verified] + +| Platform | File | `worker_cls` | dist backend | Notes | +| --- | --- | --- | --- | --- | +| CUDA | [vllm/platforms/cuda.py](../../vllm/vllm/platforms/cuda.py) | `vllm.v1.worker.gpu_worker.Worker` ([#L325](../../vllm/vllm/platforms/cuda.py#L325)) | nccl | dtype: bf16 needs CC≥8.0, fp16 needs CC≥6.0; `is_integrated_gpu` via `torch.cuda.get_device_properties(...).is_integrated` ([#L714](../../vllm/vllm/platforms/cuda.py#L714)) | +| ROCm | [vllm/platforms/rocm.py](../../vllm/vllm/platforms/rocm.py) | `vllm.v1.worker.gpu_worker.Worker` ([#L916](../../vllm/vllm/platforms/rocm.py#L916)) | nccl (rccl) | shares the GPU worker; AITER/Triton attention backends | +| XPU | [vllm/platforms/xpu.py](../../vllm/vllm/platforms/xpu.py) | `vllm.v1.worker.xpu_worker.XPUWorker` ([#L363](../../vllm/vllm/platforms/xpu.py#L363)) | xccl | own worker + `xpu_model_runner.py` | +| CPU | [vllm/platforms/cpu.py](../../vllm/vllm/platforms/cpu.py) | `vllm.v1.worker.cpu_worker.CPUWorker` ([#L282](../../vllm/vllm/platforms/cpu.py#L282)) | gloo | see below | +| Zen CPU | [vllm/platforms/zen_cpu.py](../../vllm/vllm/platforms/zen_cpu.py) | inherits CPU | gloo | bf16/fp32 only, zentorch GEMM | +| TPU | [vllm/platforms/tpu.py](../../vllm/vllm/platforms/tpu.py) | (in `tpu_inference`) | — | shim only; `tpu_input_batch.py` is a leftover | +| OOT | via plugin | plugin-defined | plugin-defined | e.g. Ascend, Metal, Gaudi (not in tree) | + +**Memory allocation / pools.** [Verified] GPU worker profiles memory once (`determine_available_memory` → `model_runner.profile_run()` under `memory_profiling`, honoring `gpu_memory_utilization` or an explicit `kv_cache_memory_bytes`) ([gpu_worker.py#L525](../../vllm/vllm/v1/worker/gpu_worker.py#L525)); `EngineCore._initialize_kv_caches` then computes `KVCacheConfig` (block count, groups, layout) and calls `initialize_from_config` + `compile_or_warm_up_model` ([core.py#L254](../../vllm/vllm/v1/engine/core.py#L254)). Sleep mode / weight offload use a custom CUDA-VMM allocator (`cumem_allocator` extension, [csrc/cumem_allocator.cpp](../../vllm/csrc/cumem_allocator.cpp), [vllm/device_allocator/cumem.py](../../vllm/vllm/device_allocator/cumem.py)); XPU has an analogous `xpumem.py`. CUDA graphs share one global graph memory pool (`Platform.get_global_graph_pool`). CPU: KV space is `VLLM_CPU_KVCACHE_SPACE` (GiB) or `kv_cache_memory_bytes`, plain `torch` CPU tensors; `tcmalloc` and PyTorch's `libgomp` are force-`LD_PRELOAD`ed on Linux x86/ARM/s390x ([cpu.py#L369](../../vllm/vllm/platforms/cpu.py#L369)). + +**Host↔device transfers and streams.** [Verified] The runner keeps `CpuGpuBuffer` pairs (pinned host + device) and copies inputs per step; async output copy uses a dedicated stream (`_get_or_create_async_output_copy_stream`, [gpu_model_runner.py#L1181](../../vllm/vllm/v1/worker/gpu_model_runner.py#L1181)). MRV2 assumes "the core model execution loop is a CUDA stream with no CPU synchronization points" ([docs/design/model_runner_v2.md](../../vllm/docs/design/model_runner_v2.md)). CPU has no streams; `_set_torch_accelerator_to_noop` stubs stream/sync calls and `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1` is forced ([cpu_model_runner.py#L244](../../vllm/vllm/v1/worker/cpu_model_runner.py#L244), [cpu.py#L356](../../vllm/vllm/platforms/cpu.py#L356)). Collectives go through `DeviceCommunicatorBase` subclasses: `cuda_communicator` (NCCL, custom all-reduce, quick-reduce, symm-mem), `cpu_communicator` (gloo + shared-memory `shm_allreduce` ops from `csrc/cpu/shm.cpp`), `xpu_communicator`, `ray_communicator` ([vllm/distributed/device_communicators/](../../vllm/vllm/distributed/device_communicators/base_device_communicator.py)). + +**CPU platform specifics (ARM edge target).** [Verified] +- `supported_dtypes`: x86/aarch64 → bf16/fp16/fp32; macOS ARM → fp16/fp32 (+bf16 only if `sysctl hw.optional.arm.FEAT_BF16` = 1); RISC-V → bf16/fp16/fp32; POWER → bf16/fp32/fp16 ([cpu.py#L107](../../vllm/vllm/platforms/cpu.py#L107)). +- `check_and_update_config` forces: `async_scheduling=False`, `mp` executor (OpenMP needs it), `cudagraph_capture_sizes=[]`, compilation mode `DYNAMO_TRACE_ONCE` with Inductor (`eager` only under `VLLM_CPU_CI_ENV`), compilation disabled if LoRA, `+gelu/+gelu_tanh/+gelu_and_mul` custom ops enabled on ARM, `spawn` start method, `TORCHINDUCTOR_COMPILE_THREADS=1`, MLA on non-AMX disables chunked prefill/prefix caching ([cpu.py#L198](../../vllm/vllm/platforms/cpu.py#L198)). +- Kernel import: x86 loads `_C`, `_C_AVX512` or `_C_AVX2` depending on runtime flags; every other arch loads `_C` ([cpu.py#L553](../../vllm/vllm/platforms/cpu.py#L553)). +- Attention ISA: `amx` (x86 AMX + bf16 + block%32) → `neon` (ARM, block%32, "NEON FMLA and BFMMLA (bf16)") → `rvv` → `vxe` → `vsx` → `vec`/`vec16` ([cpu_attn.py#L481](../../vllm/vllm/v1/attention/backends/cpu_attn.py#L481)); implementations live in [csrc/cpu/cpu_attn_neon.hpp](../../vllm/csrc/cpu/cpu_attn_neon.hpp), [csrc/cpu/cpu_attn_neon_bfmmla.hpp](../../vllm/csrc/cpu/cpu_attn_neon_bfmmla.hpp), [csrc/cpu/cpu_types_arm.hpp](../../vllm/csrc/cpu/cpu_types_arm.hpp). +- NUMA/thread binding: `CPUWorker.__init__` binds memory nodes and OpenMP threads (`VLLM_CPU_OMP_THREADS_BIND=auto|nobind|list`, `VLLM_CPU_NUM_OF_RESERVED_CPU`, `CPU_VISIBLE_MEMORY_NODES`) ([cpu_worker.py#L33](../../vllm/vllm/v1/worker/cpu_worker.py#L33), [docs/getting_started/installation/cpu.md](../../vllm/docs/getting_started/installation/cpu.md#L150)). +- Pipeline/tensor parallel on CPU use gloo + shm ops; docs recommend TP = number of NUMA nodes ([cpu.md](../../vllm/docs/getting_started/installation/cpu.md)). + +CPU-related environment variables ([vllm/envs.py#L877](../../vllm/vllm/envs.py#L877), [docs/getting_started/installation/cpu.md#L150](../../vllm/docs/getting_started/installation/cpu.md#L150)) [Verified]: + +| Variable | Default | Effect | +| --- | --- | --- | +| `VLLM_TARGET_DEVICE` | `cuda` | Build/select target (`cpu` forces CPU platform even on GPU hosts) | +| `VLLM_CPU_KVCACHE_SPACE` | unset (0) | KV cache size in GiB (legacy; `kv_cache_memory_bytes` preferred) | +| `VLLM_CPU_OMP_THREADS_BIND` | `auto` | Core list / `auto` (per-NUMA) / `nobind` for OpenMP threads | +| `VLLM_CPU_NUM_OF_RESERVED_CPU` | unset | Cores left free for the frontend per rank | +| `CPU_VISIBLE_MEMORY_NODES` | unset | NUMA node mask, like `CUDA_VISIBLE_DEVICES` | +| `VLLM_CPU_ATTN_SPLIT_KV` | `1` | Split-KV attention on CPU | +| `VLLM_CPU_INT4_W4A8` | `1` | Enable dynamic 4-bit W4A8 path | +| `VLLM_ZENTORCH_WEIGHT_PREPACK` | `1` | Zen: prepack linear weights for ZenDNN | +| `VLLM_CPU_CI_ENV` | `0` | Internal: eager instead of Inductor | +| `VLLM_ENABLE_V1_MULTIPROCESSING` | `1` | `0` = in-process engine core (CPU still forces the `mp` executor) | +| `VLLM_USE_V2_MODEL_RUNNER` | unset | Force V1/V2 model runner (auto: V2 if Triton present and no unsupported features) | +| `VLLM_PLUGINS` | unset | Allow-list of plugin names to load | + +**Portability matrix (what is actually in tree).** [Verified] Linux x86_64 (CUDA/ROCm/XPU/CPU), Linux aarch64 (CUDA e.g. GH200 CI job, CPU), Linux ppc64le/s390x/riscv64 (CPU), macOS arm64 (CPU only), WSL2 (CUDA/ROCm detection). [Verified] No Windows native support code; no Android/iOS bindings; no Vulkan/OpenCL/Metal/QNN/NNAPI kernels. CUDA arch list depends on nvcc: `7.5;8.0;8.6;8.7;8.9;9.0;10.0;(10.7);11.0;12.0` for CUDA ≥13.0 — `8.7` = Jetson Orin, `11.0` = Thor ([CMakeLists.txt#L112](../../vllm/CMakeLists.txt#L112)). ROCm arch list includes `gfx1150/1151/1152/1153` (RDNA3.5 APUs) ([CMakeLists.txt#L52](../../vllm/CMakeLists.txt#L52)). + +## 3. Kernel layer + +**Operator set and registration.** [Verified] Three C++ extension families: +- CUDA `_C_stable_libtorch` (132 `def(` ops in [csrc/libtorch_stable/torch_bindings.cpp](../../vllm/csrc/libtorch_stable/torch_bindings.cpp), registered with `STABLE_TORCH_LIBRARY` so the wheel is ABI-stable across torch minors) plus `_moe_C_stable_libtorch`; legacy `_C` remains ROCm-only ("CUDA ops migrated to `_C_stable_libtorch`", [CMakeLists.txt#L367](../../vllm/CMakeLists.txt#L367)). Contents: activations, layernorm(+quant fusion), RoPE, fused QK-norm+RoPE, cache kernels, `merge_attn_states`, MLA helpers, sampler/top-k, MoE align/topk/permute, Marlin/Machete/CUTLASS/AWQ/GPTQ/FP4/FP8/INT8 GEMMs, custom all-reduce. +- CPU `_C` (73 `def(` ops in [csrc/cpu/torch_bindings.cpp](../../vllm/csrc/cpu/torch_bindings.cpp#L357)): `silu_and_mul`, `gelu_*`, `rms_norm`, `fused_add_rms_norm`, `rotary_embedding`, `onednn_mm`/`onednn_scaled_mm`, `static/dynamic_scaled_int8_quant`, `fp8_scaled_mm_cpu`, `weight_packed_linear`, `fused_experts_cpu`, `cpu_fused_moe`, `causal_conv1d_*_cpu`, `decode/extend_attention_cpu`, `shm_allreduce/gather`, `dynamic_4bit_int_moe`, plus ARM-only (`ARM_I8MM_SUPPORT && ARM_BF16_SUPPORT && !__APPLE__`) `prepack_moe_weight_int8`/`cpu_fused_moe_int8` ([torch_bindings.cpp#L690](../../vllm/csrc/cpu/torch_bindings.cpp#L690)). Micro-GEMMs per ISA in [csrc/cpu/micro_gemm/](../../vllm/csrc/cpu/micro_gemm/cpu_micro_gemm_neon.hpp) (`amx`, `neon`, `int8_neon`, `rvv`, `vec`, `vsx`). +- Externally fetched kernel projects (CUDA only): vllm-flash-attn, FlashMLA, FlashKDA, MSA/fmha_sm100, tml-fa4, qutlass, DeepGEMM, Triton kernels ([cmake/external_projects/](../../vllm/cmake/external_projects/vllm_flash_attn.cmake)). Pip-side kernel deps: `flashinfer-python`, `tilelang`, `nvidia-cutlass-dsl`, `quack-kernels`, `humming-kernels`, `tokenspeed-mla` ([requirements/cuda.txt](../../vllm/requirements/cuda.txt)). + +Python-side Triton kernels are spread across `vllm/model_executor/layers/**` (fused MoE, Mamba, RoPE, etc.), `vllm/v1/sample/ops/topk_topp_triton.py`, `vllm/v1/worker/gpu/*`, and [vllm/kernels/triton/](../../vllm/vllm/kernels/triton/qkv_padded_fp8_quant.py); Helion and "oink" kernels exist under [vllm/kernels/](../../vllm/vllm/kernels/oink_ops.py). + +**Dispatch mechanism.** [Verified] Two coexisting mechanisms: +1. `CustomOp` (nn.Module subclasses such as `RMSNorm`, `SiluAndMul`, `RotaryEmbedding`): `dispatch_forward()` picks `forward_hip/cpu/tpu/xpu/oot/cuda` by `current_platform`, or `forward_native` (pure torch, optionally `torch.compile`d) when the op is disabled. Enablement is per-op via `CompilationConfig.custom_ops` (`all`/`none` + `+op`/`-op`); with Inductor the default is `none`, so Inductor generates fused Triton for them ([custom_op.py#L174](../../vllm/vllm/model_executor/custom_op.py#L174), [docs/design/custom_op.md](../../vllm/docs/design/custom_op.md)). OOT plugins replace an in-tree class by name with `@CustomOp.register_oot(name=...)` / `@PluggableLayer.register_oot` — instantiation of the in-tree class returns the OOT subclass ([custom_op.py#L103](../../vllm/vllm/model_executor/custom_op.py#L103)). +2. `vllm.ir` (newer): `@register_op` declares a functional op (`rms_norm`, `fused_add_rms_norm` today, [vllm/ir/ops/layernorm.py](../../vllm/vllm/ir/ops/layernorm.py)); providers register implementations with `op.register_impl("vllm_c", supported=..., supports_args=...)`; `IrOp.dispatch` walks a per-op **priority list** (platform default e.g. `["vllm_c","native"]` when not compiling, `["native"]` under Inductor, `["oink", ...]` if `VLLM_USE_OINK_OPS`) and returns the first impl whose `supports_args` passes ([ir/op.py#L244](../../vllm/vllm/ir/op.py#L244), [cuda.py#L723](../../vllm/vllm/platforms/cuda.py#L723), [docs/design/vllm_ir.md](../../vllm/docs/design/vllm_ir.md)). +3. Attention: `AttentionBackendEnum` maps names to class paths; `register_backend` can override any member (including `CUSTOM`) at runtime; the platform's `get_attn_backend_cls` ranks candidates by device capability, head size, dtype, block size, sliding window, etc. via `AttentionBackend.validate_configuration` ([registry.py](../../vllm/vllm/v1/attention/backends/registry.py), [cuda.py#L434](../../vllm/vllm/platforms/cuda.py#L434), [backend.py#L270](../../vllm/vllm/v1/attention/backend.py#L270)). In-tree backends: FLASH_ATTN(+DIFFKV), TRITON_ATTN(+DIFFKV), FLASHINFER, FLEX_ATTENTION, ROCM_ATTN/AITER variants, ~12 MLA variants, HPC_ATTN, B12X, TURBOQUANT, NO_ATTENTION, CPU_ATTN, CPU_MLA, AMX_MLA; Mamba-family backends MAMBA1/MAMBA2/SHORT_CONV/LINEAR/GDN_ATTN. +Default attention backend candidates per platform [Verified]: + +| Platform | Candidate order (non-MLA, decoder) | Source | +| --- | --- | --- | +| CUDA CC 10.x (Blackwell) | FLASHINFER → FLASH_ATTN → TRITON_ATTN → FLEX_ATTENTION | [cuda.py#L157](../../vllm/vllm/platforms/cuda.py#L157) | +| CUDA other (≥8.0) | FLASH_ATTN → FLASHINFER → TRITON_ATTN → FLEX_ATTENTION | [cuda.py#L165](../../vllm/vllm/platforms/cuda.py#L165) | +| CUDA MLA (CC 9.0/10.x/12.x) | FLASHINFER_MLA / FLASH_ATTN_MLA / CUTLASS_MLA / FLASHMLA variants by CC | [cuda.py#L85](../../vllm/vllm/platforms/cuda.py#L85) | +| ROCm | ROCM_AITER_UNIFIED_ATTN / ROCM_AITER_FA / ROCM_ATTN → TRITON_ATTN → TURBOQUANT; MLA: ROCM_AITER_MLA / TRITON_MLA | [rocm.py#L463](../../vllm/vllm/platforms/rocm.py#L463) | +| XPU | FLASH_ATTN (SYCL) / TRITON_ATTN; MLA: TRITON_MLA / XPU_MLA_SPARSE | [xpu.py#L145](../../vllm/vllm/platforms/xpu.py#L145) | +| CPU | CPU_ATTN (ISA-dispatched); MLA: AMX_MLA (x86 AMX) or CPU_MLA | [cpu.py#L138](../../vllm/vllm/platforms/cpu.py#L138) | +| ViT encoders | FLASH_ATTN / TRITON_ATTN / FLASHINFER / TORCH_SDPA by platform | [cuda.py#L536](../../vllm/vllm/platforms/cuda.py#L536) | + +4. Linear/quant kernels: `choose_mp_linear_kernel` iterates `MPLinearKernel` subclasses and picks the first whose `can_implement(config)` succeeds ([compressed_tensors_w4a8_int.py#L104](../../vllm/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a8_int.py#L104)); CPU unquantized GEMM is chosen at load time by `dispatch_cpu_unquantized_gemm`: zentorch → sgl-kernel `weight_packed_linear` (AMX only, weight ≤1 MiB) → oneDNN `onednn_mm` (non-POWER) → `torch.nn.functional.linear` ([layers/utils.py#L434](../../vllm/vllm/model_executor/layers/utils.py#L434)). + +**Quantization formats and where dequant happens.** [Verified] `QUANTIZATION_METHODS`: awq, auto_awq, fp8, fbgemm_fp8 (deprecated), fp_quant (deprecated), modelopt, modelopt_fp4, modelopt_mxfp8, modelopt_mixed, auto_gptq, gptq, gptq_marlin, awq_marlin, humming, compressed-tensors, experts_int8, quark, moe_wna16, torchao, inc, mxfp4, gpt_oss_mxfp4, deepseek_v4_fp8, online (+ online sub-modes fp8_per_tensor/per_block/per_channel, int8_per_channel_weight_only, nvfp4_per_token, mxfp8) ([quantization/__init__.py#L16](../../vllm/vllm/model_executor/layers/quantization/__init__.py#L16)). Third-party configs can be added with `register_quantization_config`. Weight-only formats are dequantized **inside fused GEMM kernels** (Marlin/Machete/CUTLASS W4A8, oneDNN int8 on CPU, KleidiAI dynamic-4-bit on ARM via `torch.ops.aten._dyn_quant_matmul_4bit`, [dynamic_4bit.py#L14](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py#L14)); activation quant (fp8/int8) is fused into RMSNorm/SiLU kernels where available (`fused_layernorm_dynamic_per_token_quant`, `fused_silu_mul_block_quant`). KV cache dtypes: `auto, float16, bfloat16, fp8, fp8_e4m3, fp8_e5m2, fp8_inc, fp8_ds_mla, nvfp4_ds_mla, turboquant_*, int4/int8/fp8_per_token_head, nvfp4, nvfp4_4over6` ([vllm/config/cache.py#L39](../../vllm/vllm/config/cache.py#L39)); on CPU FP8 KV requires x86 AVX-512/AMX ([cpu_attn.py#L500](../../vllm/vllm/v1/attention/backends/cpu_attn.py#L500)). [Verified] Repo-reported hardware/format matrix for Arm CPU: only llm-compressor INT8 W8A8 and W4A8 marked supported; AWQ/GPTQ/Marlin/GGUF marked unsupported ([docs/features/quantization/README.md#L50](../../vllm/docs/features/quantization/README.md#L50)). GGUF and bitsandbytes now require `vllm-gguf-plugin` / `vllm-bnb-plugin` ([gguf.md](../../vllm/docs/features/quantization/gguf.md), [bnb.md](../../vllm/docs/features/quantization/bnb.md)). + +**Compilation / JIT / graph capture.** [Verified] `CompilationMode`: `NONE`, `STOCK_TORCH_COMPILE`, `DYNAMO_TRACE_ONCE`, `VLLM_COMPILE` (piecewise: model is split at attention/`splitting_ops`, each piece compiled by Inductor with guard dropping, artifacts cached under `~/.cache/vllm/torch_compile_cache`) ([config/compilation.py#L37](../../vllm/vllm/config/compilation.py#L37), [docs/design/torch_compile.md](../../vllm/docs/design/torch_compile.md)). `CUDAGraphMode`: `NONE, PIECEWISE, FULL, FULL_DECODE_ONLY, FULL_AND_PIECEWISE`; `CudagraphDispatcher.dispatch` maps a `BatchDescriptor` (padded size, uniform-decode flag, LoRA case) to a captured graph or falls back to eager ([cudagraph_dispatcher.py#L235](../../vllm/vllm/v1/cudagraph_dispatcher.py#L235)). Optimization levels `-O0..-O3` set these defaults ([docs/design/optimization_levels.md](../../vllm/docs/design/optimization_levels.md)). Warmup: `Worker.compile_or_warm_up_model` runs `_dummy_run` for compile sizes, `kernel_warmup`, `capture_model()` (largest→smallest), then a sampler dummy run; all before the first request ([gpu_worker.py#L762](../../vllm/vllm/v1/worker/gpu_worker.py#L762)). ViT encoders have their own CUDA-graph path ([vllm/v1/worker/encoder_cudagraph.py](../../vllm/vllm/v1/worker/encoder_cudagraph.py), [docs/design/cuda_graphs_multimodal.md](../../vllm/docs/design/cuda_graphs_multimodal.md)). CPU uses `DYNAMO_TRACE_ONCE` + Inductor C++ codegen (no graphs) ([cpu.py#L292](../../vllm/vllm/platforms/cpu.py#L292)). + +**Fallback when an op is unsupported.** [Verified] `CustomOp` → `forward_native` (pure PyTorch). `vllm.ir` → the last provider in the priority list must accept all args (normally `native`). Attention → candidate list ordered by platform; if none validates, an error lists why each was rejected ([cuda.py#L434](../../vllm/vllm/platforms/cuda.py#L434)). CPU without Triton-CPU → `_postprocess_triton` monkey-patches Triton kernels (slot mapping, EAGLE/DFlash input prep, rejection sampler, Mamba memcpy) with C++/torch equivalents ([cpu_model_runner.py#L65](../../vllm/vllm/v1/worker/cpu_model_runner.py#L65)); with Triton-CPU present the patches are skipped. Missing compiled extension modules only log a warning at import (`import_kernels`) but any op call then fails. + +**How to add a custom op / backend (extension points).** [Verified] +- New platform: package with `entry_points={"vllm.platform_plugins": [...]}` returning a `Platform` subclass path; implement `check_and_update_config` (set `worker_cls`), `get_attn_backend_cls`, `get_device_communicator_cls`; a `WorkerBase` subclass; an `AttentionBackend`; optional OOT `CustomOp`s. Reference skeleton: [tests/plugins/vllm_add_dummy_platform/](../../vllm/tests/plugins/vllm_add_dummy_platform/setup.py) with `dummy_platform.py`, `dummy_attention_backend.py`, `dummy_custom_ops.py` ([docs/design/plugin_system.md](../../vllm/docs/design/plugin_system.md)). +- New op impl: `@ir.ops..register_impl("provider", supported=..., supports_args=...)` or a `CustomOp.forward_`; C++ ops via `torch.ops` (see `vllm/_custom_ops.py`). +- New attention backend: `register_backend(AttentionBackendEnum.CUSTOM, "pkg.mod.Class")`. +- New model: `ModelRegistry.register_model("Arch", "pkg.mod:Class")` from a `vllm.general_plugins` entry point ([docs/contributing/model/registration.md](../../vllm/docs/contributing/model/registration.md)). +- New quant method: `register_quantization_config`. +- Plugin loading is gated by `VLLM_PLUGINS` and happens in every process ([vllm/plugins/__init__.py](../../vllm/vllm/plugins/__init__.py)). [Verified] Triton ops cannot be overridden via the OOT mechanism ("Custom way doesn't work for triton ops now", plugin doc). + +## 4. Model runner + +**Model loading / formats.** [Verified] Models are HF-format directories loaded by `ModelRegistry` → architecture class → `DefaultModelLoader` (safetensors/pt/npcache), or alternative loaders: `dummy`, `fastsafetensors`, `instanttensor`, `ipc_cache`, `mistral`, `modelexpress`, `runai_streamer(_sharded)`, `sharded_state`, `tensorizer` ([vllm/model_executor/model_loader/__init__.py#L33](../../vllm/vllm/model_executor/model_loader/__init__.py#L33), [vllm/config/load.py](../../vllm/vllm/config/load.py)). There is **no export/conversion pipeline**: weights are mapped in-place into vLLM's parallel layers (`AutoWeightsLoader`, `WeightsMapper`); quantized checkpoints are consumed in their vendor formats. A `transformers` fallback backend (`_TRANSFORMERS_BACKEND_MODELS`) runs unsupported HF architectures ([registry.py#L698](../../vllm/vllm/model_executor/models/registry.py#L698)). Weight reload / RL weight-update RPCs exist (`reload_weights`, `update_weights`, [gpu_worker.py#L1391](../../vllm/vllm/v1/worker/gpu_worker.py#L1391)). + +**Execution lifecycle.** [Verified] +1. `EngineCore.__init__` → `Executor` spawns workers → `Worker.init_device` (device select, dist init, memory snapshot, `init_workspace_manager`, choose V1 vs V2 runner) ([gpu_worker.py#L357](../../vllm/vllm/v1/worker/gpu_worker.py#L357)) → `load_model`. +2. `_initialize_kv_caches`: `register_all_kvcache_specs`, gather per-layer `KVCacheSpec`s, resolve `KVCacheLayout`, `determine_available_memory` (profile run), `get_kv_cache_configs`, possibly auto-fit `max_model_len`, `initialize_from_config` (allocate KV tensors, bind to attention layers), `compile_or_warm_up_model` (compile + CUDA graphs + sampler warmup) ([core.py#L254](../../vllm/vllm/v1/engine/core.py#L254)). +3. Steady state: `EngineCore.step` → `scheduler.schedule()` → `executor.execute_model(scheduler_output, non_block=True)` → grammar bitmask → `sample_tokens` → `scheduler.update_from_output` ([core.py#L608](../../vllm/vllm/v1/engine/core.py#L608)). With pipeline parallel or async scheduling `step_with_batch_queue` keeps up to N in-flight batches. +4. Worker step (V1 runner): `_update_states` (apply scheduler diffs to the persistent batch) → `_execute_mm_encoder` (encoder-only inputs, cached by `mm_hash`) → `_prepare_inputs` (token ids, positions, slot mappings, attention metadata per KV group, spec-decode metadata) → `_determine_batch_execution_and_padding` (DP/micro-batch/CUDA-graph descriptor) → `set_forward_context` → `_model_forward` → `compute_logits` → `_sample` → `_bookkeeping_sync` (async copy to host) ([gpu_model_runner.py#L4274](../../vllm/vllm/v1/worker/gpu_model_runner.py#L4274)). + +Lifecycle summary [Verified]: + +| Phase | Where | What happens | +| --- | --- | --- | +| Config resolution | [vllm/config/vllm.py](../../vllm/vllm/config/vllm.py) + `Platform.check_and_update_config` | Optimization level → compile/cudagraph modes; async scheduling; worker class; CPU overrides | +| Process spawn | `EngineCoreClient.make_client` → `EngineCoreProc.run_engine_core`; `Executor._init_executor` | Engine core and worker processes, ZMQ handshake (`startup_handshake`), shm queues | +| Device init | `Worker.init_device` / `CPUWorker.init_device` | Device select, `init_worker_distributed_environment`, memory snapshot, workspace manager, runner construction | +| Load | `Worker.load_model` → `GPUModelRunner.load_model` | Loader by `load_format`, weight mapping, `process_weights_after_loading` (quant repack, CPU GEMM dispatch) | +| Profile | `determine_available_memory` → `profile_run` | Dummy max batch (incl. dummy MM inputs) to size KV cache | +| KV alloc | `EngineCore._initialize_kv_caches` → `initialize_from_config` → `initialize_kv_cache` | Group specs, block count, layout, tensor allocation, bind to attention layers | +| Warmup | `compile_or_warm_up_model` | `_dummy_run` per compile size, `kernel_warmup`, `capture_model` (CUDA graphs), sampler dummy run | +| Serve | `run_busy_loop` → `step` / `step_with_batch_queue` | Schedule → execute → sample → update; outputs pushed to the client socket | +| Idle / sleep | `sleep`, `wake_up`, `pause_generation` | Free weights/KV via cumem allocator, keep process alive | +| Shutdown | `EngineCore.shutdown`, `Executor.shutdown` | Abort in-flight requests, join workers | + +**Dynamic shapes.** [Verified] Batches are token-flattened (`num_tokens` varies) with padding to the nearest CUDA-graph capture size or `compile_sizes`/`compile_ranges`; Dynamo marks the token dimension dynamic and vLLM drops guards ([docs/design/torch_compile.md](../../vllm/docs/design/torch_compile.md)). `max_num_seqs`/`max_num_batched_tokens` bound persistent buffers. MRV2 pre-allocates `max_num_reqs` rows (1024 default) and assigns a fixed row per request ([docs/design/model_runner_v2.md](../../vllm/docs/design/model_runner_v2.md)). + +**State & KV cache.** [Verified] Allocation: KV tensors are allocated per `KVCacheGroupSpec` at `initialize_kv_cache` ([gpu_model_runner.py#L7491](../../vllm/vllm/v1/worker/gpu_model_runner.py#L7491)); layout chosen from six `KVCacheLayout` permutations (`LBHNC`, `LBNHC`, `LHBNC`, `BLHNC`, `BLNHC`, `BHLNC`) per backend support ([kv_cache_layout.py](../../vllm/vllm/v1/kv_cache_layout.py)); CPU backend packs its own layout (`CpuPlatform.pack_kv_cache`). Paging: fixed `block_size` (backend-preferred; hybrid models align attention and Mamba page sizes, [interface.py#L609](../../vllm/vllm/platforms/interface.py#L609)); `BlockPool` holds all blocks plus `BlockHashToBlockMap` for prefix hits; `FreeKVCacheBlockQueue` is an O(1) doubly linked LRU list, ordered so the tail of a freed chain is evicted first ([kv_cache_utils.py#L230](../../vllm/vllm/v1/core/kv_cache_utils.py#L230)). Prefix caching: block hash = `hash(parent_hash, block_tokens, extra: lora_id, mm_hashes, cache_salt)`; default `sha256`, options `sha256_cbor`, `xxhash`, `xxhash_cbor`; only full blocks are cached; partial-tail caching exists (`cache_partial_block`) ([docs/design/prefix_caching.md](../../vllm/docs/design/prefix_caching.md), [block_pool.py#L225](../../vllm/vllm/v1/core/block_pool.py#L225)). Reuse across calls: `get_computed_blocks` at admission returns the longest cached prefix per group; sliding-window/chunked-local/Mamba managers implement their own `find_longest_cache_hit` and `remove_skipped_blocks` ([single_type_kv_cache_manager.py](../../vllm/vllm/v1/core/single_type_kv_cache_manager.py)). Eviction to host: `vllm/v1/kv_offload` (CPU offload with policies, Triton swap kernels) and KV connectors (NIXL, LMCache, Mooncake, MoRIIO, FlexKV, offloading/simple CPU offload) for disaggregated prefill ([vllm/v1/kv_offload/cpu/manager.py](../../vllm/vllm/v1/kv_offload/cpu/manager.py), [vllm/distributed/kv_transfer/kv_connector/v1/base.py](../../vllm/vllm/distributed/kv_transfer/kv_connector/v1/base.py#L171)). Encoder outputs have a separate `EncoderCacheManager` budget ([vllm/v1/core/encoder_cache_manager.py](../../vllm/vllm/v1/core/encoder_cache_manager.py)). Session/streaming requests keep KV alive between turns (`WAITING_FOR_STREAMING_REQ`, `_update_request_as_session`, [scheduler.py#L1545](../../vllm/vllm/v1/core/sched/scheduler.py#L1545)). + +**Multimodal stages.** [Verified] Preprocessing runs in the API-server process (`BaseMultiModalProcessor` wraps the HF processor; `PromptInsertion/Replacement` expand placeholder tokens; outputs are hashed and cached in `MultiModalProcessorCache` variants including a shared-memory object store between API server and engine) ([processing/processor.py#L1030](../../vllm/vllm/multimodal/processing/processor.py#L1030), [cache.py](../../vllm/vllm/multimodal/cache.py)). Encoders (ViT, Whisper encoder, audio towers) run inside the worker step under an encoder token budget and are cached per `mm_hash`; embeddings are scattered into `inputs_embeds` before the LM forward (`_execute_mm_encoder`, `_gather_mm_embeddings`, [gpu_model_runner.py#L3066](../../vllm/vllm/v1/worker/gpu_model_runner.py#L3066)). Encoder–decoder (Whisper) uses `CrossAttentionSpec` KV groups and `EncDecMultiModalProcessor`. Disaggregated encoder execution via EC connectors ([docs/features/disagg_encoder.md](../../vllm/docs/features/disagg_encoder.md), [vllm/v1/worker/mm_encoder_model_runner.py](../../vllm/vllm/v1/worker/mm_encoder_model_runner.py)). [Verified] No vocoder/decoder/diffusion *output* stages exist; `DiffusionGemma` implements block-diffusion **text** decoding with a custom `DiffusionSampler` and `ModelState` ([diffusion_gemma.py#L1043](../../vllm/vllm/model_executor/models/diffusion_gemma.py#L1043)). + +## 5. Scheduler + +### (a) Request / token / pipeline-level scheduling + +[Verified] `Scheduler.schedule()` ([scheduler.py#L535](../../vllm/vllm/v1/core/sched/scheduler.py#L535)) — the code comment states there is "no decoding phase nor prefill phase": each request has `num_computed_tokens` and `num_tokens_with_spec`; the loop assigns up to `token_budget = max_num_scheduled_tokens` (defaults to `max_num_batched_tokens`) new tokens per step. +- **Running requests first**: for each, `num_new_tokens = num_tokens_with_spec - num_computed_tokens`, capped by `long_prefill_token_threshold` (chunked prefill), token budget, `max_model_len`, Mamba block alignment, encoder budget (`_try_schedule_encoder_inputs`), multi-module MTP lookahead; `kv_cache_manager.allocate_slots` reserves blocks; on failure, **preemption**: the lowest-priority (priority policy) or last (FCFS) running request is popped, its blocks freed, `num_computed_tokens=0`, and it is prepended to `waiting` (recompute, no swap-out) ([scheduler.py#L1452](../../vllm/vllm/v1/core/sched/scheduler.py#L1452)). +- **Waiting requests** are then admitted while budget remains and `len(running) < max_num_seqs`; prefix-cache hits reduce their initial `num_new_tokens`; requests may be parked as `WAITING_FOR_REMOTE_KVS`, `WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR`, or `WAITING_FOR_STREAMING_REQ` ([vllm/v1/request.py#L364](../../vllm/vllm/v1/request.py#L364)). +- **Batching**: continuous batching with mixed prefill/decode in one `SchedulerOutput`; `SchedulingPolicy.FCFS | PRIORITY` via `create_request_queue` ([request_queue.py](../../vllm/vllm/v1/core/sched/request_queue.py)); DP "prefill throttling" for cadence alignment (`throttle_prefills`). +- **Speculative decoding** reserves `draft_slots` and passes `scheduled_spec_decode_tokens`; drafts come from EAGLE/EAGLE3/MTP/Medusa/DFlash/n-gram/suffix proposers ([vllm/v1/spec_decode/](../../vllm/vllm/v1/spec_decode/eagle.py)). +- **Async scheduling**: `AsyncScheduler` overrides `_update_after_schedule`/`_update_request_with_output` to schedule step N+1 with placeholder output tokens before step N's sampled ids are known ([async_scheduler.py](../../vllm/vllm/v1/core/sched/async_scheduler.py)); default is **on** unless pooling model, non-EAGLE/n-gram spec-decode, executor lacks support, ROCm DeepEP-DBO, or CPU platform ([config/vllm.py#L1273](../../vllm/vllm/config/vllm.py#L1273), [cpu.py#L268](../../vllm/vllm/platforms/cpu.py#L268)). +- **Pipeline/multi-stage**: PP is handled by `step_with_batch_queue` (batch queue depth = PP stages) ([core.py#L649](../../vllm/vllm/v1/engine/core.py#L649)); multi-model "stages" (e.g. thinker→talker) are **not** a scheduler concept — N/A. +- **Cancellation/abort**: `AsyncLLM.abort` → `EngineCoreClient.abort_requests_async` → `EngineCoreProc` aborts queue → `Scheduler.finish_requests(FINISHED_ABORTED)`; blocks are freed (possibly deferred, `_drain_deferred_frees`) ([async_llm.py#L834](../../vllm/vllm/v1/engine/async_llm.py#L834), [scheduler.py#L2483](../../vllm/vllm/v1/core/sched/scheduler.py#L2483)). Client disconnects propagate through the FastAPI handlers to `abort`. +- **Backpressure**: `AsyncLLM.check_admission` enforces max in-flight requests/tokens ([async_llm.py#L288](../../vllm/vllm/v1/engine/async_llm.py#L288)); `pause_generation/resume_generation` and `PauseState` stop scheduling (used for sleep/weight update); `max_num_seqs`, `max_num_batched_tokens`, `stream_interval` (token buffering) are the tuning knobs ([config/scheduler.py#L190](../../vllm/vllm/config/scheduler.py#L190)). + +Scheduler knobs and where they bite [Verified]: + +| Knob (`SchedulerConfig` / `CacheConfig`) | Effect in `Scheduler` | +| --- | --- | +| `max_num_seqs` | `max_num_running_reqs`: cap on RUNNING set; waiting admission stops at it | +| `max_num_batched_tokens` / `max_num_scheduled_tokens` | Per-step `token_budget` | +| `long_prefill_token_threshold` | Chunk size for long prompts (chunked prefill) | +| `enable_chunked_prefill` | Disabled automatically for non-causal (prefix-LM) layers and CPU MLA without AMX | +| `policy` = `fcfs` / `priority` | Queue type and preemption victim selection | +| `async_scheduling` | `AsyncScheduler` subclass; auto-resolved in `VllmConfig` | +| `stream_interval` | Token batching for streaming output | +| `enable_prefix_caching`, `prefix_caching_hash_algo` | `KVCacheManager.get_computed_blocks`, `BlockPool` hashing | +| `block_size`, `kv_cache_memory_bytes`, `gpu_memory_utilization` | Block geometry and pool size | +| `max_num_encoder_input_tokens` (derived) | Encoder compute budget per step | +| `speculative_config.max_num_new_slots_for_drafting` | `draft_slots` reserved from `input_budget` | + +Concrete `SchedulerOutput` contents ([vllm/v1/core/sched/output.py](../../vllm/vllm/v1/core/sched/output.py)): new request descriptors (prompt ids, MM features, block ids), cached-request diffs (new tokens, new block ids, resumed flags), `num_scheduled_tokens` per request, `scheduled_spec_decode_tokens`, `scheduled_encoder_inputs`, `free_encoder_mm_hashes`, `num_common_prefix_blocks` (cascade attention), `preempted_req_ids`, structured-output flags (`has_structured_output_requests`, `pending_structured_output_tokens`), `kv_connector_metadata`, and `finished_req_ids` ([output.py#L223](../../vllm/vllm/v1/core/sched/output.py#L223)). The worker never sees request objects — only this diff, which is what makes the persistent-batch runner possible. + +### (b) Graph / operator / thread-level scheduling + +[Verified] There is **no operator-level scheduler**: op ordering is whatever the PyTorch module graph (eager or Dynamo-captured) produces, with Inductor fusion passes ([vllm/compilation/passes/](../../vllm/vllm/compilation/passes/pass_manager.py)). Concurrency knobs at this layer: +- CUDA graph replay chosen per batch by `CudagraphDispatcher` ([cudagraph_dispatcher.py](../../vllm/vllm/v1/cudagraph_dispatcher.py)). +- Micro-batching / Dual-Batch-Overlap (`enable_dbo`, `ubatch_slices`, `gpu_ubatch_wrapper.py`) splits a batch into two streams to overlap communication and compute ([vllm/v1/worker/gpu_ubatch_wrapper.py](../../vllm/vllm/v1/worker/gpu_ubatch_wrapper.py), [docs/design/dbo.md](../../vllm/docs/design/dbo.md)); disabled on CPU. +- Separate streams: async output copy stream; shared-experts stream in MoE (disabled on CPU). +- Thread pools: CPU backend binds one OpenMP thread per core per TP rank (`VLLM_CPU_OMP_THREADS_BIND`) and forbids `torch.set_num_threads` after binding ([cpu_worker.py#L120](../../vllm/vllm/v1/worker/cpu_worker.py#L120)); `TORCHINDUCTOR_CPP_DYNAMIC_THREADS=1`. Engine core uses dedicated input/output socket threads ([core.py#L1705](../../vllm/vllm/v1/engine/core.py#L1705)); structured-output bitmask filling is submitted to a thread pool ([vllm/v1/structured_output/__init__.py](../../vllm/vllm/v1/structured_output/__init__.py)). +- Stream assignment per op / explicit op scheduling: **N/A**. + +## 6. I/O layer + +- **Text preprocessing / tokenization.** [Verified] `Renderer` classes turn chat/completion prompts into token ids and MM data (`hf`, `mistral`, `cohere`, `deepseek_v32/v4`, `kimi_k3`, `inkling`, `terratorch`) ([vllm/renderers/registry.py](../../vllm/vllm/renderers/registry.py), [vllm/renderers/base.py](../../vllm/vllm/renderers/base.py)). Tokenizer modes: HF `tokenizers` (fast, via `CachedHfTokenizer`), Mistral `mistral_common`, custom (`kimi_audio`, DeepSeek encodings) ([vllm/tokenizers/registry.py](../../vllm/vllm/tokenizers/registry.py)); `fastokens` (optional Rust package) patches HF fast tokenizers ([vllm/tokenizers/fastokens.py](../../vllm/vllm/tokenizers/fastokens.py)); `sentencepiece` and `tiktoken` are dependencies ([requirements/common.txt](../../vllm/requirements/common.txt)). Prompt types: `TextPrompt`, `TokensPrompt`, `EmbedsPrompt` (prompt embeddings), `ExplicitEncoderDecoderPrompt`, `DataPrompt` ([vllm/inputs/llm.py](../../vllm/vllm/inputs/llm.py)). +- **Image/video/audio preprocessing.** [Verified] `vllm/multimodal/media/{image,video,audio}.py` load bytes/base64/URL; images via Pillow, video via OpenCV/PyAV/torchcodec/PyNvVideoCodec backends ([vllm/multimodal/video.py](../../vllm/vllm/multimodal/video.py)), audio via PyAV decode with resampling by `soxr`/`scipy`/PyAV and mono downmix (`AudioResampler`, `normalize_audio`) ([vllm/multimodal/audio.py](../../vllm/vllm/multimodal/audio.py)); optional `audio` extra = `av, scipy, soundfile, soxr, mistral_common[audio]` ([setup.py#L1515](../../vllm/setup.py#L1515)). `split_audio`/`find_split_point` chunk long audio at quiet points for ASR ([audio.py#L325](../../vllm/vllm/multimodal/audio.py#L325)). +- **Streaming.** [Verified] Token streaming: `RequestOutputCollector` (async queue per request) → SSE in the OpenAI routers; `stream_interval` batches tokens ([output_processor.py#L48](../../vllm/vllm/v1/engine/output_processor.py#L48)). Streaming **input**: `AsyncLLM.generate` accepts an async generator of `StreamingInput` (prompt chunks + sampling params) for multi-turn sessions that keep KV; used by the realtime ASR path ([async_llm.py#L515](../../vllm/vllm/v1/engine/async_llm.py#L515), [vllm/engine/protocol.py#L30](../../vllm/vllm/engine/protocol.py#L30), [tests/v1/streaming_input/](../../vllm/tests/v1/streaming_input/test_async_llm_streaming.py)). Realtime audio: WebSocket `/v1/realtime` (`input_audio_buffer.append/commit`, `transcription.delta/done` events) → `RealtimeConnection.audio_stream_generator` → model's `buffer_realtime_audio` → streaming prompts ([speech_to_text/realtime/connection.py](../../vllm/vllm/entrypoints/speech_to_text/realtime/connection.py), [docs/serving/online_serving/speech_to_text.md#L169](../../vllm/docs/serving/online_serving/speech_to_text.md#L169)). No audio *output* streaming exists. +- **Postprocessing.** [Verified] Incremental detokenization (`FastIncrementalDetokenizer` using `tokenizers.DecodeStream`, slow fallback), stop-string checking, logprobs, reasoning/tool parsers ([vllm/v1/engine/detokenizer.py](../../vllm/vllm/v1/engine/detokenizer.py), [vllm/reasoning/](../../vllm/vllm/reasoning/__init__.py), [vllm/tool_parsers/](../../vllm/vllm/tool_parsers/__init__.py)); pooling outputs (embedding/classification/score) ([vllm/outputs.py](../../vllm/vllm/outputs.py)); ASR post-processing (`post_process_output`, segment timestamps, diarization) in `SupportsTranscription`. IO-processor plugins allow custom pre/post for pooling models ([vllm/plugins/io_processors/interface.py](../../vllm/vllm/plugins/io_processors/interface.py)). +- **Application boundaries.** [Verified] Python: `LLM` (offline: `generate/chat/encode/embed/score/classify`, `apply_model`, `collective_rpc`) and `AsyncLLM`. HTTP: OpenAI-compatible FastAPI (`/v1/completions`, `/v1/chat/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/audio/transcriptions`, `/v1/audio/translations`, `/v1/realtime`, scoring, tokenize, Anthropic `/v1/messages`, Cohere), endpoint plugins ([docs/serving/online_serving/openai_compatible_server.md](../../vllm/docs/serving/online_serving/openai_compatible_server.md), [vllm/entrypoints/anthropic/](../../vllm/vllm/entrypoints/anthropic/__init__.py)). gRPC: `vllm.entrypoints.grpc_server` backed by the external `smg-grpc-servicer` package ([grpc_server.py](../../vllm/vllm/entrypoints/grpc_server.py)). CLI: `vllm serve|chat|complete|bench|run-batch|collect-env` ([vllm/entrypoints/cli/main.py](../../vllm/vllm/entrypoints/cli/main.py)). Rust: experimental drop-in frontend (`vllm-server` axum, tokenizer, chat templates) talking ZMQ to the Python engine core ([rust/README.md](../../vllm/rust/README.md)). **No C API, no mobile bindings.** + +## 7. Execution flow + +### (a) Text LLM decode (online, V1 runner, single GPU) + +```mermaid +sequenceDiagram + participant C as HTTP client + participant API as FastAPI router (api server proc) + participant A as AsyncLLM + participant IP as InputProcessor/Renderer + participant CC as AsyncMPClient (ZMQ) + participant EC as EngineCoreProc (busy loop) + participant S as Scheduler + KVCacheManager + participant EX as MultiprocExecutor + participant W as Worker / GPUModelRunner + participant OP as OutputProcessor + C->>API: POST /v1/chat/completions (stream=true) + API->>A: generate(prompt, SamplingParams) + A->>IP: render chat template, tokenize, validate lens + A->>CC: add_request_async(EngineCoreRequest) + CC-->>EC: msgspec over ZMQ input socket + EC->>S: add_request → waiting queue + loop each step (run_busy_loop → step) + EC->>S: schedule(): get_computed_blocks (prefix hit), allocate_slots, token budget + S-->>EC: SchedulerOutput (new/cached reqs, num_scheduled_tokens, block ids) + EC->>EX: execute_model(scheduler_output, non_block=True) + EX-->>W: shm MessageQueue broadcast + W->>W: _update_states, _prepare_inputs, attn metadata + W->>W: CudagraphDispatcher.dispatch(BatchDescriptor) + W->>W: model forward (compiled pieces / CUDA graph), compute_logits + EC->>EX: sample_tokens(grammar_bitmask) + W->>W: Sampler (temp, penalties, top-k/p), async copy to host + W-->>EC: ModelRunnerOutput (sampled ids, logprobs) + EC->>S: update_from_output → append tokens, cache_blocks, finish checks + EC-->>CC: EngineCoreOutputs via ZMQ output socket + CC->>OP: process_outputs → IncrementalDetokenizer, stop strings + OP-->>A: RequestOutputCollector.put(RequestOutput delta) + A-->>API: async iterator yields delta + API-->>C: SSE chunk + end + C--xAPI: disconnect / cancel + API->>A: abort(request_id) → EngineCore abort → scheduler frees blocks +``` + +### (b) Realtime streaming ASR (audio in → text out) — the closest TTS/VLA-relevant path + +```mermaid +sequenceDiagram + participant C as WS client (mic) + participant R as RealtimeConnection (/v1/realtime) + participant SV as OpenAIServingRealtime + participant M as Model cls (SupportsRealtime, e.g. VoxtralRealtimeGeneration) + participant A as AsyncLLM (streaming input session) + participant S as Scheduler + participant W as GPUModelRunner + C->>R: session.update, input_audio_buffer.append (base64 PCM) ... + R->>R: audio_stream_generator yields np.ndarray chunks + R->>SV: transcribe_realtime(audio_stream) + SV->>M: buffer_realtime_audio(audio_stream, input_stream, model_config) + M-->>A: async generator of StreamingInput prompts (audio segment + prefix tokens) + A->>A: _add_streaming_input_request (session keeps KV between chunks) + loop per audio segment + A->>S: request enters WAITING_FOR_STREAMING_REQ → RUNNING when chunk arrives + S->>W: schedule encoder input (audio encoder budget) + LM tokens + W->>W: _execute_mm_encoder (audio tower) → embeddings → LM forward → sample (realtime_max_tokens) + W-->>S: sampled ids + S-->>A: EngineCoreOutputs + A-->>SV: RequestOutput delta + SV-->>R: transcription.delta + R-->>C: WS event + end + C->>R: input_audio_buffer.commit → transcription.done +``` + +[Verified] For OpenVLA the flow is exactly (a) with an image: `OpenVLAMultiModalProcessor` → DINOv2+SigLIP `PrismaticVisionBackbone` → `PrismaticProjector` → Llama decode of `n_action_bins` action tokens ([openvla.py#L377](../../vllm/vllm/model_executor/models/openvla.py#L377)); un-normalizing action tokens into continuous actions is left to the client (not in tree). TTS: **N/A** (no path). + +## 8. Tests and examples + +- [Verified] Tests: `tests/` (~60 top-level areas): `tests/v1/{core,engine,worker,e2e,attention,cudagraph,kv_connector,kv_offload,sample,spec_decode,structured_output,streaming_input,...}` ([tests/v1/core/test_scheduler.py](../../vllm/tests/v1/core/test_scheduler.py), [tests/v1/core/test_prefix_caching.py](../../vllm/tests/v1/core/test_prefix_caching.py), [tests/v1/engine/test_engine_core.py](../../vllm/tests/v1/engine/test_engine_core.py)); kernels in `tests/kernels/{attention,core,mamba,moe,quantization,ir,helion}` incl. [tests/kernels/test_onednn.py](../../vllm/tests/kernels/test_onednn.py); models in `tests/models` (e.g. [tests/models/multimodal/processing/test_openvla.py](../../vllm/tests/models/multimodal/processing/test_openvla.py)); platform test [tests/platforms/test_cpu.py](../../vllm/tests/platforms/test_cpu.py); plugin fixtures in [tests/plugins/](../../vllm/tests/plugins/vllm_add_dummy_platform/setup.py) and [tests/plugins_tests/](../../vllm/tests/plugins_tests/test_endpoint_plugins.py) (gguf/bnb plugin tests live here). Evals under `tests/evals`. +- [Verified] How to run (per [AGENTS.md](../../vllm/AGENTS.md)): `uv venv --python 3.12`, `VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto`, `uv pip install -r requirements/test/cuda.in`, then `.venv/bin/python -m pytest tests/path/to/test_file.py -v`; never bare `pip`. Model-specific test guidance: [docs/contributing/model/tests.md](../../vllm/docs/contributing/model/tests.md). +- [Verified] CI: Buildkite pipelines ([.buildkite/test-pipeline.yaml](../../vllm/.buildkite/test-pipeline.yaml), hardware jobs for AMD, Ascend NPU, **CPU x86 + "Arm CPU Test Shard" via `run-cpu-test-arm.sh`**, GH200 (aarch64 + Hopper), Intel/XPU ([.buildkite/hardware_tests/cpu.yaml#L149](../../vllm/.buildkite/hardware_tests/cpu.yaml#L149), [.buildkite/hardware_tests/gh200.yaml](../../vllm/.buildkite/hardware_tests/gh200.yaml), [.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh](../../vllm/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh)); GitHub Actions macOS Apple-Silicon smoke test ([.github/workflows/macos-smoke-test.yml](../../vllm/.github/workflows/macos-smoke-test.yml)); wheel size guard 500 MB ([.buildkite/check-wheel-size.py](../../vllm/.buildkite/check-wheel-size.py)); CPU performance benchmark JSONs referenced from the CPU docs. +- [Verified] Examples: basic offline/online ([examples/basic/offline_inference/basic.py](../../vllm/examples/basic/offline_inference/basic.py)); vision/audio ([examples/generate/multimodal/audio_language_offline.py](../../vllm/examples/generate/multimodal/audio_language_offline.py), [vision_language_offline.py](../../vllm/examples/generate/multimodal/vision_language_offline.py), [encoder_decoder_multimodal_offline.py](../../vllm/examples/generate/multimodal/encoder_decoder_multimodal_offline.py)); Qwen2.5-Omni **thinker only** ([examples/generate/multimodal/qwen2_5_omni/only_thinker.py](../../vllm/examples/generate/multimodal/qwen2_5_omni/only_thinker.py)); ASR clients ([examples/speech_to_text/openai/openai_transcription_client.py](../../vllm/examples/speech_to_text/openai/openai_transcription_client.py)); realtime microphone Gradio demo ([examples/speech_to_text/realtime/openai_realtime_microphone_client.py](../../vllm/examples/speech_to_text/realtime/openai_realtime_microphone_client.py)); disaggregated / RL / Ray serving. **No mobile, embedded, or TTS examples.** + +## 9. TTS relevance + +- [Verified] **Models producing audio: none.** Search for vocoders/codec decoders/audio-output endpoints returns only audio *encoder* code (`mimo_audio.py` holds MiMo's RVQ audio tokenizer/encoder used as an input tower by `MiMoV2OmniForCausalLM`; Qwen2.5/3-Omni are registered as **Thinker** halves with `"talker.": None` weight-skip; the example README is titled "Thinker Only") ([mimo_audio.py](../../vllm/vllm/model_executor/models/mimo_audio.py), [qwen2_5_omni_thinker.py#L1062](../../vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py#L1062), [examples/generate/multimodal/qwen2_5_omni/README.md](../../vllm/examples/generate/multimodal/qwen2_5_omni/README.md)). +- [Verified] **Audio input models** (registry `_MULTIMODAL_MODELS`): AudioFlamingo3, FunAudioChat, Gemma3n, Granite Speech(+Plus), Kimi-Audio, MiDashengLM, MiMo-V2-Omni, MiniCPM-O, MOSS-Audio, MossTranscribeDiarize, Phi-4-MM, Qwen2-Audio, Qwen2.5-Omni thinker, Qwen3-Omni thinker, Qwen3-ASR, Ultravox, Voxtral, Voxtral Realtime, Whisper, VibeVoice-ASR, GLM-ASR, FireRedASR2, FunASR, Cohere-Transcribe ([registry.py#L345](../../vllm/vllm/model_executor/models/registry.py#L345), [docs/models/supported_models.md#L706](../../vllm/docs/models/supported_models.md#L706)). +- [Verified] **Audio ops**: resampling (soxr/scipy/PyAV), mono downmix, normalization, duration, quiet-point chunking ([vllm/multimodal/audio.py](../../vllm/vllm/multimodal/audio.py)); Whisper-style encoder attention via `MMEncoderAttention`; sinusoidal position embeddings; RVQ codebooks (encoder side only). No STFT/iSTFT, no mel-inverse, no HiFi-GAN/BigVGAN/WavTokenizer decoder, no flow-matching/DiT audio decoder. +- [Verified] **Streaming audio**: input-side only (`SupportsRealtime.buffer_realtime_audio`, `VoxtralRealtimeBuffer` with look-ahead/look-back ms, `realtime_max_tokens` per segment) ([voxtral_realtime.py#L127](../../vllm/vllm/model_executor/models/voxtral_realtime.py#L127), [interfaces.py#L1267](../../vllm/vllm/model_executor/models/interfaces.py#L1267)). +- [Verified] **Interfaces to build on**: `SupportsTranscription` + `SpeechToTextBaseServing` show how an audio-modal endpoint, chunking, language detection, and streaming post-processors are wired ([docs/contributing/model/transcription.md](../../vllm/docs/contributing/model/transcription.md)). Streaming-input sessions (`StreamingInput`, `WAITING_FOR_STREAMING_REQ`) give a KV-preserving multi-turn primitive that a text→speech-token model could reuse. +- **What is missing for TTS** [Verified unless noted]: (1) any speech-token → waveform stage and a place to run it (no "post-LM decoder stage" in the runner); (2) audio output types in `RequestOutput`/OpenAI `/v1/audio/speech`; (3) non-text sampling constraints for codec-token vocabularies (would have to be logits processors); (4) [Inferred] multi-stage pipeline scheduling (thinker→talker→vocoder) — the scheduler has one model per engine; vLLM-Omni is the project's answer, not this repo. Registering a TTS LM whose output is codec tokens is feasible as a plain text-generation model, but decoding to audio must happen outside vLLM. [Proposal] For edge TTS, use vLLM only if the LM half dominates cost and a separate lightweight vocoder can run on the same device. + +## 10. VLA relevance + +- [Verified] **Vision encoders**: many ViT/SigLIP/CLIP/DINOv2/InternViT/Qwen-VL towers with their own CUDA-graph and attention-backend selection (`get_vit_attn_backend`: FLASH_ATTN/TRITON/FLASHINFER/TORCH_SDPA), `SupportsEncoderCudaGraph`, `SupportsMultiModalPruning` ([interfaces.py#L1856](../../vllm/vllm/model_executor/models/interfaces.py#L1856), [docs/design/cuda_graphs_multimodal.md](../../vllm/docs/design/cuda_graphs_multimodal.md)). +- [Verified] **Action model**: `OpenVLAForActionPrediction` — Prismatic backbone (`timm` DINOv2 + SigLIP fused), 2/3-layer MLP projector, Llama LM, `n_action_bins` from `OpenVLAConfig`, image-only modality (`get_placeholder_str` raises for non-image), 224×224 fixed patches; docs list it as supported with PP ([openvla.py](../../vllm/vllm/model_executor/models/openvla.py), [vllm/transformers_utils/configs/openvla.py](../../vllm/vllm/transformers_utils/configs/openvla.py), [docs/models/supported_models.md#L606](../../vllm/docs/models/supported_models.md#L606)). +- [Verified] **Structured/proprioceptive inputs**: none as a modality; only `EmbedsPrompt` (raw prompt embeddings) could carry precomputed state embeddings ([vllm/inputs/llm.py#L131](../../vllm/vllm/inputs/llm.py#L131), [docs/features/prompt_embeds.md](../../vllm/docs/features/prompt_embeds.md)). +- [Verified] **Action heads / diffusion / flow policies**: none. `DiffusionGemma` proves the runner can host a non-AR block-diffusion decoder with a custom sampler and per-request state (`DiffusionGemmaRequestStates`, `ModelState`) — a template for iterative denoising heads over the LM's KV cache, but it produces text tokens ([diffusion_gemma.py#L660](../../vllm/vllm/model_executor/models/diffusion_gemma.py#L660)). +- [Verified] **Low-latency loop primitives**: async scheduling, full CUDA graphs for uniform decode, `FULL_DECODE_ONLY`, MRV2's sync-free stream loop, priority scheduling, streaming input sessions that keep KV (`session_id`) across turns, `stream_interval`. [Inferred] The 3-process architecture (API server ↔ engine core ↔ worker over ZMQ/shm) adds fixed IPC latency per step that a single-process control loop would avoid; `InprocClient` + `UniprocExecutor` (`VLLM_ENABLE_V1_MULTIPROCESSING=0`) reduce this to one process on GPU, but CPU forces the `mp` executor ([cpu.py#L273](../../vllm/vllm/platforms/cpu.py#L273)). +- **Missing for VLA** [Verified]: proprioception/state tokens, continuous action heads (MLP/diffusion/flow), action chunk decoding, KV reuse across timesteps for a fixed instruction with changing images (prefix caching covers the text prefix only; image embeddings are cached by `mm_hash` so identical frames hit, new frames do not), and any robotics I/O. + +## 11. Edge deployment profile + +- **Build system.** [Verified] `setuptools` + CMake (`cmake>=3.26.1`, `ninja`), `setuptools-rust` and a Rust `1.95` toolchain ([rust-toolchain.toml](../../vllm/rust-toolchain.toml)) for the optional Rust frontend/`fastokens` (precompiled artifacts can be reused: `VLLM_USE_PRECOMPILED`, `VLLM_USE_PRECOMPILED_RUST`) ([setup.py#L49](../../vllm/setup.py#L49), [build_rust.sh](../../vllm/build_rust.sh)). Target selection: `VLLM_TARGET_DEVICE=cuda|rocm|xpu|cpu|empty` (macOS forced to `cpu`) ([setup.py#L85](../../vllm/setup.py#L85)). CPU build: [cmake/cpu_extension.cmake](../../vllm/cmake/cpu_extension.cmake) detects ISA from `/proc/cpuinfo`/`sysctl` (`asimd`, `bf16`, `i8mm`, AVX2/512/AMX, POWER, S390 VXE, RVV), sets `-march=armv8.2-a+bf16+dotprod+fp16` (or without bf16), builds **oneDNN** (pinned commit for aarch64) and **Arm Compute Library** via SCons (`ONEDNN_AARCH64_USE_ACL`, `VLLM_USE_ACL`; `ACL_ROOT_DIR` to reuse a prebuilt one), and C++20. Cross-compile flags: `VLLM_CPU_ARM_BF16`, `VLLM_CPU_ARM_I8MM`, `VLLM_CPU_RVV_BF16`. Apple Silicon skips oneDNN/ACL (`APPLE_SILICON_FOUND`). +- **Dependencies (runtime).** [Verified] `torch==2.13.0` (+cpu on Linux x86/aarch64/s390x), `transformers>=5.10.4`, `tokenizers`, `safetensors`, `fastapi[standard]`, `uvicorn` (via fastapi standard), `pyzmq`, `msgspec`, `numba==0.65.0`, `opencv-python-headless`, `pillow`, `torchaudio`, `torchvision`, `torchcodec`, `xgrammar`, `llguidance`, `outlines_core`, `lm-format-enforcer`, `compressed-tensors`, `mistral_common`, `openai`, `anthropic`, `mcp`, `opentelemetry-*`, `prometheus_client`, `intel-openmp` (x86 CPU), `py-cpuinfo` ([requirements/common.txt](../../vllm/requirements/common.txt), [requirements/cpu.txt](../../vllm/requirements/cpu.txt)). [Inferred] This is a multi-GB Python environment even on CPU; there is no "minimal" profile. +- **Binary size.** [Verified] CI fails if a wheel exceeds `VLLM_MAX_SIZE_MB` (default 500 MB) ([.buildkite/check-wheel-size.py#L12](../../vllm/.buildkite/check-wheel-size.py#L12)). [Unknown] Actual CPU/aarch64 wheel size and RSS were not measured here. +- **Process model.** [Verified] API server + engine core + N workers (+DP coordinator), `spawn`/`forkserver` start methods, ZMQ sockets, shared-memory `MessageQueue`; single-process mode via `VLLM_ENABLE_V1_MULTIPROCESSING=0` (GPU only in practice) ([docs/design/arch_overview.md#L67](../../vllm/docs/design/arch_overview.md#L67), [docs/design/multiprocessing.md](../../vllm/docs/design/multiprocessing.md)). +- **Supported platforms for edge** [Verified]: + - *NVIDIA Jetson/Orin/Thor*: detection path exists (`cuda_is_jetson`, non-NVML platform), arch `8.7` and `11.0` in `CUDA_SUPPORTED_ARCHS`; docs only contain a Thor `sm_110a` Triton `ptxas` workaround (`TRITON_PTXAS_PATH`); no Jetson wheels/Docker/CI in tree; kernel deps (FlashInfer, FA, CUTLASS DSL) would need aarch64+CUDA builds — the GH200 CI job shows aarch64+CUDA is built ([__init__.py#L76](../../vllm/vllm/platforms/__init__.py#L76), [troubleshooting.md](../../vllm/docs/usage/troubleshooting.md)). `is_integrated_gpu()` exists on the platform interface (used for unified-memory awareness) ([cuda.py#L714](../../vllm/vllm/platforms/cuda.py#L714)). + - *ARM CPU (Linux aarch64)*: first-class: NEON/BF16/I8MM kernels, oneDNN+ACL GEMM, KleidiAI W4A8 via torch, INT8 W8A8 (`onednn_scaled_mm`), int8 fused MoE, prebuilt `manylinux_2_34_aarch64` wheels since 0.11.2, nightly wheels per commit, `vllm/vllm-openai-cpu:latest-arm64` images, ARM CI shards, "tested on AWS Graviton3" ([cpu.arm.inc.md](../../vllm/docs/getting_started/installation/cpu.arm.inc.md), [docker/Dockerfile.cpu#L143](../../vllm/docker/Dockerfile.cpu#L143)). Requirements: Linux, gcc≥12.3, NEON; BF16 needs ARMv8.6 (`FEAT_BF16`). Triton-CPU is only built into the x86 Docker image (`VLLM_CPU_X86` guard), so on ARM the runner uses the C++ monkey-patches and **Model Runner V1** (`HAS_TRITON` false → V2 disabled) ([Dockerfile.cpu#L311](../../vllm/docker/Dockerfile.cpu#L311), [vllm/triton_utils/importing.py#L15](../../vllm/vllm/triton_utils/importing.py#L15), [config/vllm.py#L672](../../vllm/vllm/config/vllm.py#L672)). + - *Apple Silicon*: in-tree **CPU only** (experimental; FP16/FP32, BF16 if `FEAT_BF16`; build from source; no wheels/images; oneDNN/ACL disabled; CI smoke test); GPU only via OOT `vllm-metal` (MLX) ([cpu.apple.inc.md](../../vllm/docs/getting_started/installation/cpu.apple.inc.md), [gpu.apple.inc.md](../../vllm/docs/getting_started/installation/gpu.apple.inc.md)). No CoreML/ANE. + - *Android / mobile GPU-NPU (Vulkan, OpenCL, QNN, NNAPI)*: **nothing in tree**; would require an OOT platform plugin plus a PyTorch build for that target. [Inferred] Not realistic given the torch/transformers/FastAPI dependency stack. + - *Other*: AMD RDNA3/3.5 APU gfx targets are in the ROCm arch list ([CMakeLists.txt#L52](../../vllm/CMakeLists.txt#L52)); RISC-V RVV and s390x CPU builds exist. +- **Model conversion path and constraints.** [Verified] None required: HF checkpoints (safetensors) are loaded directly; quantized checkpoints must come pre-quantized in a supported format (AWQ/GPTQ/compressed-tensors/ModelOpt/etc.) or use the `online` quant method; on ARM CPU only INT8 W8A8 and dynamic 4-bit W4A8 (compressed-tensors `W4A8Int` with `Dynamic4bitLinearKernel`, groupwise bf16 scales, KleidiAI-supported shapes) have kernels ([dynamic_4bit.py#L26](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py#L26)). `torch.compile` cache directory can be copied to a target to avoid compile time ([docs/design/torch_compile.md](../../vllm/docs/design/torch_compile.md)). [Verified] CPU-validated model list is Intel Xeon-centric ([docs/models/hardware_supported_models/cpu.md](../../vllm/docs/models/hardware_supported_models/cpu.md)); no ARM-specific validated list. + +## 12. Limitations and unknowns + +- [Verified] No TTS / audio generation of any kind; no vocoder or codec-decoder ops; no `/v1/audio/speech`. +- [Verified] Only one VLA model (OpenVLA), text-token action outputs only; no proprioceptive modality or continuous action heads. +- [Verified] Hard dependencies on PyTorch 2.13, transformers ≥5.10, FastAPI/uvicorn, ZMQ; no C API, no mobile bindings, no Android/Vulkan/OpenCL/QNN/NNAPI/CoreML support. +- [Verified] TPU, Ascend NPU, Apple Metal, Gaudi, GGUF, bitsandbytes all live in out-of-tree packages; compatibility of those plugins with this HEAD is outside this tree ([Unknown]). +- [Verified] CPU platform disables async scheduling, DBO, CUDA-graph-equivalents; compilation is `DYNAMO_TRACE_ONCE` + Inductor (C++ codegen), LoRA disables compilation; MLA on non-AMX CPUs disables chunked prefill and prefix caching. +- [Verified] Model Runner V2 requires Triton; on ARM CPU (no Triton-CPU in the ARM image) V1 runner is used, and several Triton kernels are replaced by C++ fallbacks (`_postprocess_triton`). +- [Verified] Arm CPU quantization coverage per docs: INT8 W8A8 and W4A8 only; AWQ/GPTQ/Marlin/GGUF marked unsupported. +- [Verified] Apple Silicon in-tree is CPU-only, experimental, source-build only, no prebuilt wheels. +- [Verified] Jetson: detection only; no wheels, Docker, CI, or docs beyond a Thor ptxas workaround. +- [Verified] `bitsandbytes`/`DeepSpeedFP` rows still appear in the quantization support table although BnB is now a plugin — docs lag code. +- [Unknown] Binary/wheel size, RSS, startup time and latency on ARM/Jetson — not measured; the repo publishes no edge numbers (CPU benchmark dashboard referenced in docs is Xeon-based). +- [Unknown] Whether `vllm-metal`, `vllm-ascend`, or other OOT platforms track this exact HEAD (platform interface has had removals: `use_v1`, `_Backend`, `seed_everything`, `validate_request(prompt)` per the plugin doc's deprecation list). +- [Inferred] Minimum practical footprint is a server-class Python environment; trimming to an embedded profile would mean forking the executor/worker layer (e.g. `InprocClient` + `UniprocExecutor` + a slim frontend) and dropping the HTTP stack. +- [Verified] The installed `vllm 0.28.0` wheel on this machine differs from the analyzed checkout (post-0.29.0rc4 main); behaviours described here (e.g. MRV2 default, `vllm.ir`, `_C_stable_libtorch`, streaming input sessions) may be absent or different in 0.28.0. + +## 13. Reference index + +| Path | Role | +| --- | --- | +| [AGENTS.md](../../vllm/AGENTS.md) | Repo contribution/dev workflow (uv, tests, lint) | +| [pyproject.toml](../../vllm/pyproject.toml) | Build requirements (cmake, ninja, setuptools-rust, torch==2.13.0), Python 3.10–3.14 | +| [setup.py](../../vllm/setup.py) | Target-device selection, CMake extension build, precompiled wheel logic, extras (`zen`, `audio`, `tensorizer`, `runai`) | +| [CMakeLists.txt](../../vllm/CMakeLists.txt) | CUDA/HIP arch lists, extension targets (`_C_stable_libtorch`, `_moe_C_stable_libtorch`, `cumem_allocator`, `_C`, `_rocm_C`, `spinloop`, `fs_io_C`) | +| [cmake/cpu_extension.cmake](../../vllm/cmake/cpu_extension.cmake) | CPU ISA detection, ARM flags, oneDNN + ACL build, CPU `_C` sources | +| [cmake/external_projects/vllm_flash_attn.cmake](../../vllm/cmake/external_projects/vllm_flash_attn.cmake) | Example of FetchContent kernel dependency | +| [requirements/common.txt](../../vllm/requirements/common.txt), [requirements/cpu.txt](../../vllm/requirements/cpu.txt), [requirements/cuda.txt](../../vllm/requirements/cuda.txt) | Runtime dependency sets | +| [rust-toolchain.toml](../../vllm/rust-toolchain.toml), [build_rust.sh](../../vllm/build_rust.sh), [rust/README.md](../../vllm/rust/README.md) | Experimental Rust frontend | +| [docker/Dockerfile.cpu](../../vllm/docker/Dockerfile.cpu) | x86/arm64 CPU image; Triton-CPU only on x86 | +| [csrc/libtorch_stable/torch_bindings.cpp](../../vllm/csrc/libtorch_stable/torch_bindings.cpp) | CUDA op registrations (STABLE_TORCH_LIBRARY) | +| [csrc/cpu/torch_bindings.cpp](../../vllm/csrc/cpu/torch_bindings.cpp) | CPU op registrations incl. ARM I8MM-guarded MoE ops | +| [csrc/cpu/cpu_attn_neon.hpp](../../vllm/csrc/cpu/cpu_attn_neon.hpp), [csrc/cpu/cpu_attn_neon_bfmmla.hpp](../../vllm/csrc/cpu/cpu_attn_neon_bfmmla.hpp), [csrc/cpu/cpu_types_arm.hpp](../../vllm/csrc/cpu/cpu_types_arm.hpp) | ARM NEON/BFMMLA attention and vector types | +| [csrc/cpu/micro_gemm/cpu_micro_gemm_neon.hpp](../../vllm/csrc/cpu/micro_gemm/cpu_micro_gemm_neon.hpp) | ARM micro-GEMM | +| [csrc/cpu/dnnl_kernels.cpp](../../vllm/csrc/cpu/dnnl_kernels.cpp) | oneDNN GEMM / int8 quant kernels | +| [csrc/cumem_allocator.cpp](../../vllm/csrc/cumem_allocator.cpp) | CUDA VMM allocator for sleep mode | +| [vllm/platforms/__init__.py](../../vllm/vllm/platforms/__init__.py) | Platform detection and plugin resolution | +| [vllm/platforms/interface.py](../../vllm/vllm/platforms/interface.py) | `Platform` ABC, enums | +| [vllm/platforms/cuda.py](../../vllm/vllm/platforms/cuda.py), [vllm/platforms/rocm.py](../../vllm/vllm/platforms/rocm.py), [vllm/platforms/xpu.py](../../vllm/vllm/platforms/xpu.py), [vllm/platforms/cpu.py](../../vllm/vllm/platforms/cpu.py), [vllm/platforms/zen_cpu.py](../../vllm/vllm/platforms/zen_cpu.py), [vllm/platforms/tpu.py](../../vllm/vllm/platforms/tpu.py) | In-tree platforms | +| [vllm/plugins/__init__.py](../../vllm/vllm/plugins/__init__.py) | Entry-point plugin groups | +| [vllm/v1/engine/async_llm.py](../../vllm/vllm/v1/engine/async_llm.py), [vllm/v1/engine/llm_engine.py](../../vllm/vllm/v1/engine/llm_engine.py) | Engine front-ends | +| [vllm/v1/engine/core.py](../../vllm/vllm/v1/engine/core.py), [vllm/v1/engine/core_client.py](../../vllm/vllm/v1/engine/core_client.py) | Engine core process and clients | +| [vllm/v1/engine/input_processor.py](../../vllm/vllm/v1/engine/input_processor.py), [vllm/v1/engine/output_processor.py](../../vllm/vllm/v1/engine/output_processor.py), [vllm/v1/engine/detokenizer.py](../../vllm/vllm/v1/engine/detokenizer.py) | Input validation, output streaming, detokenization | +| [vllm/engine/protocol.py](../../vllm/vllm/engine/protocol.py) | `EngineClient`, `StreamingInput` | +| [vllm/v1/core/sched/scheduler.py](../../vllm/vllm/v1/core/sched/scheduler.py), [vllm/v1/core/sched/async_scheduler.py](../../vllm/vllm/v1/core/sched/async_scheduler.py), [vllm/v1/core/sched/interface.py](../../vllm/vllm/v1/core/sched/interface.py), [vllm/v1/core/sched/request_queue.py](../../vllm/vllm/v1/core/sched/request_queue.py) | Request scheduler | +| [vllm/v1/core/kv_cache_manager.py](../../vllm/vllm/v1/core/kv_cache_manager.py), [vllm/v1/core/kv_cache_coordinator.py](../../vllm/vllm/v1/core/kv_cache_coordinator.py), [vllm/v1/core/single_type_kv_cache_manager.py](../../vllm/vllm/v1/core/single_type_kv_cache_manager.py), [vllm/v1/core/block_pool.py](../../vllm/vllm/v1/core/block_pool.py), [vllm/v1/core/kv_cache_utils.py](../../vllm/vllm/v1/core/kv_cache_utils.py), [vllm/v1/core/encoder_cache_manager.py](../../vllm/vllm/v1/core/encoder_cache_manager.py) | KV / encoder cache management | +| [vllm/v1/kv_cache_interface.py](../../vllm/vllm/v1/kv_cache_interface.py), [vllm/v1/kv_cache_layout.py](../../vllm/vllm/v1/kv_cache_layout.py) | KV specs and layouts | +| [vllm/v1/kv_offload/cpu/manager.py](../../vllm/vllm/v1/kv_offload/cpu/manager.py), [vllm/distributed/kv_transfer/kv_connector/v1/base.py](../../vllm/vllm/distributed/kv_transfer/kv_connector/v1/base.py) | KV offload / connectors | +| [vllm/v1/request.py](../../vllm/vllm/v1/request.py) | `Request`, `RequestStatus` | +| [vllm/v1/executor/abstract.py](../../vllm/vllm/v1/executor/abstract.py), [vllm/v1/executor/multiproc_executor.py](../../vllm/vllm/v1/executor/multiproc_executor.py), [vllm/v1/executor/uniproc_executor.py](../../vllm/vllm/v1/executor/uniproc_executor.py) | Executors | +| [vllm/v1/worker/worker_base.py](../../vllm/vllm/v1/worker/worker_base.py), [vllm/v1/worker/gpu_worker.py](../../vllm/vllm/v1/worker/gpu_worker.py), [vllm/v1/worker/cpu_worker.py](../../vllm/vllm/v1/worker/cpu_worker.py), [vllm/v1/worker/xpu_worker.py](../../vllm/vllm/v1/worker/xpu_worker.py) | Workers | +| [vllm/v1/worker/gpu_model_runner.py](../../vllm/vllm/v1/worker/gpu_model_runner.py), [vllm/v1/worker/cpu_model_runner.py](../../vllm/vllm/v1/worker/cpu_model_runner.py), [vllm/v1/worker/mm_encoder_model_runner.py](../../vllm/vllm/v1/worker/mm_encoder_model_runner.py), [vllm/v1/worker/encoder_cudagraph.py](../../vllm/vllm/v1/worker/encoder_cudagraph.py), [vllm/v1/worker/gpu_ubatch_wrapper.py](../../vllm/vllm/v1/worker/gpu_ubatch_wrapper.py) | Model runner V1 and helpers | +| [vllm/v1/worker/gpu/model_runner.py](../../vllm/vllm/v1/worker/gpu/model_runner.py), [vllm/v1/worker/gpu/README.md](../../vllm/vllm/v1/worker/gpu/README.md) | Model Runner V2 | +| [vllm/v1/cudagraph_dispatcher.py](../../vllm/vllm/v1/cudagraph_dispatcher.py) | CUDA graph runtime dispatch | +| [vllm/compilation/backends.py](../../vllm/vllm/compilation/backends.py), [vllm/compilation/cuda_graph.py](../../vllm/vllm/compilation/cuda_graph.py), [vllm/compilation/decorators.py](../../vllm/vllm/compilation/decorators.py), [vllm/compilation/passes/pass_manager.py](../../vllm/vllm/compilation/passes/pass_manager.py) | torch.compile integration | +| [vllm/config/compilation.py](../../vllm/vllm/config/compilation.py), [vllm/config/scheduler.py](../../vllm/vllm/config/scheduler.py), [vllm/config/cache.py](../../vllm/vllm/config/cache.py), [vllm/config/load.py](../../vllm/vllm/config/load.py), [vllm/config/vllm.py](../../vllm/vllm/config/vllm.py) | Config dataclasses and default resolution | +| [vllm/v1/attention/backend.py](../../vllm/vllm/v1/attention/backend.py), [vllm/v1/attention/backends/registry.py](../../vllm/vllm/v1/attention/backends/registry.py), [vllm/v1/attention/selector.py](../../vllm/vllm/v1/attention/selector.py), [vllm/v1/attention/backends/cpu_attn.py](../../vllm/vllm/v1/attention/backends/cpu_attn.py) | Attention abstraction and CPU backend | +| [vllm/model_executor/custom_op.py](../../vllm/vllm/model_executor/custom_op.py) | `CustomOp` / `PluggableLayer` dispatch and OOT registration | +| [vllm/ir/op.py](../../vllm/vllm/ir/op.py), [vllm/ir/ops/layernorm.py](../../vllm/vllm/ir/ops/layernorm.py), [vllm/kernels/vllm_c.py](../../vllm/vllm/kernels/vllm_c.py), [vllm/kernels/oink_ops.py](../../vllm/vllm/kernels/oink_ops.py), [vllm/kernels/triton/qkv_padded_fp8_quant.py](../../vllm/vllm/kernels/triton/qkv_padded_fp8_quant.py) | vLLM IR ops and providers | +| [vllm/model_executor/layers/utils.py](../../vllm/vllm/model_executor/layers/utils.py) | CPU unquantized GEMM dispatch | +| [vllm/model_executor/layers/quantization/__init__.py](../../vllm/vllm/model_executor/layers/quantization/__init__.py) | Quantization method registry | +| [vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a8_int.py](../../vllm/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a8_int.py) | W4A8 int scheme (kernel selection) | +| [vllm/model_executor/kernels/linear/mixed_precision/cpu.py](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py), [vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py) | CPU WNA16 (AMX/RISC-V) and ARM KleidiAI dynamic-4bit kernels | +| [vllm/model_executor/model_loader/__init__.py](../../vllm/vllm/model_executor/model_loader/__init__.py) | Load formats | +| [vllm/model_executor/models/registry.py](../../vllm/vllm/model_executor/models/registry.py), [vllm/model_executor/models/interfaces.py](../../vllm/vllm/model_executor/models/interfaces.py) | Model registry and capability protocols | +| [vllm/model_executor/models/openvla.py](../../vllm/vllm/model_executor/models/openvla.py), [vllm/transformers_utils/configs/openvla.py](../../vllm/vllm/transformers_utils/configs/openvla.py) | OpenVLA | +| [vllm/model_executor/models/diffusion_gemma.py](../../vllm/vllm/model_executor/models/diffusion_gemma.py) | Block-diffusion text decoder + custom sampler | +| [vllm/model_executor/models/whisper.py](../../vllm/vllm/model_executor/models/whisper.py), [vllm/model_executor/models/voxtral_realtime.py](../../vllm/vllm/model_executor/models/voxtral_realtime.py), [vllm/model_executor/models/kimi_audio.py](../../vllm/vllm/model_executor/models/kimi_audio.py), [vllm/model_executor/models/moss_audio.py](../../vllm/vllm/model_executor/models/moss_audio.py), [vllm/model_executor/models/mimo_audio.py](../../vllm/vllm/model_executor/models/mimo_audio.py), [vllm/model_executor/models/qwen2_5_omni_thinker.py](../../vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py), [vllm/model_executor/models/ultravox.py](../../vllm/vllm/model_executor/models/ultravox.py), [vllm/model_executor/models/qwen2_audio.py](../../vllm/vllm/model_executor/models/qwen2_audio.py) | Audio-input models | +| [vllm/multimodal/registry.py](../../vllm/vllm/multimodal/registry.py), [vllm/multimodal/processing/processor.py](../../vllm/vllm/multimodal/processing/processor.py), [vllm/multimodal/cache.py](../../vllm/vllm/multimodal/cache.py), [vllm/multimodal/audio.py](../../vllm/vllm/multimodal/audio.py), [vllm/multimodal/media/audio.py](../../vllm/vllm/multimodal/media/audio.py), [vllm/multimodal/video.py](../../vllm/vllm/multimodal/video.py), [vllm/multimodal/encoder_budget.py](../../vllm/vllm/multimodal/encoder_budget.py) | Multimodal processing, caches, media I/O | +| [vllm/v1/sample/sampler.py](../../vllm/vllm/v1/sample/sampler.py), [vllm/v1/sample/ops/topk_topp_sampler.py](../../vllm/vllm/v1/sample/ops/topk_topp_sampler.py), [vllm/v1/sample/rejection_sampler.py](../../vllm/vllm/v1/sample/rejection_sampler.py) | Sampling | +| [vllm/v1/spec_decode/eagle.py](../../vllm/vllm/v1/spec_decode/eagle.py) | Speculative decoding proposer example | +| [vllm/v1/structured_output/__init__.py](../../vllm/vllm/v1/structured_output/__init__.py) | Grammar-constrained decoding manager | +| [vllm/lora/punica_wrapper/punica_cpu.py](../../vllm/vllm/lora/punica_wrapper/punica_cpu.py) | LoRA kernels on CPU | +| [vllm/distributed/device_communicators/base_device_communicator.py](../../vllm/vllm/distributed/device_communicators/base_device_communicator.py), [vllm/distributed/device_communicators/cpu_communicator.py](../../vllm/vllm/distributed/device_communicators/cpu_communicator.py) | Device communicators | +| [vllm/device_allocator/cumem.py](../../vllm/vllm/device_allocator/cumem.py) | Sleep-mode allocator | +| [vllm/triton_utils/importing.py](../../vllm/vllm/triton_utils/importing.py) | `HAS_TRITON` incl. Triton-CPU detection | +| [vllm/envs.py](../../vllm/vllm/envs.py) | Env vars (`VLLM_TARGET_DEVICE`, `VLLM_CPU_*`, `VLLM_USE_V2_MODEL_RUNNER`, `VLLM_PLUGINS`) | +| [vllm/entrypoints/llm.py](../../vllm/vllm/entrypoints/llm.py), [vllm/entrypoints/launchers/api_server/entry.py](../../vllm/vllm/entrypoints/launchers/api_server/entry.py), [vllm/entrypoints/cli/main.py](../../vllm/vllm/entrypoints/cli/main.py), [vllm/entrypoints/grpc_server.py](../../vllm/vllm/entrypoints/grpc_server.py), [vllm/entrypoints/anthropic/__init__.py](../../vllm/vllm/entrypoints/anthropic/__init__.py) | Application boundaries | +| [vllm/entrypoints/speech_to_text/base/serving.py](../../vllm/vllm/entrypoints/speech_to_text/base/serving.py), [vllm/entrypoints/speech_to_text/realtime/serving.py](../../vllm/vllm/entrypoints/speech_to_text/realtime/serving.py), [vllm/entrypoints/speech_to_text/realtime/connection.py](../../vllm/vllm/entrypoints/speech_to_text/realtime/connection.py) | ASR / realtime endpoints | +| [vllm/renderers/base.py](../../vllm/vllm/renderers/base.py), [vllm/renderers/registry.py](../../vllm/vllm/renderers/registry.py), [vllm/tokenizers/registry.py](../../vllm/vllm/tokenizers/registry.py), [vllm/tokenizers/fastokens.py](../../vllm/vllm/tokenizers/fastokens.py), [vllm/inputs/llm.py](../../vllm/vllm/inputs/llm.py), [vllm/outputs.py](../../vllm/vllm/outputs.py), [vllm/reasoning/__init__.py](../../vllm/vllm/reasoning/__init__.py), [vllm/tool_parsers/__init__.py](../../vllm/vllm/tool_parsers/__init__.py), [vllm/plugins/io_processors/interface.py](../../vllm/vllm/plugins/io_processors/interface.py) | I/O layer | +| [docs/design/arch_overview.md](../../vllm/docs/design/arch_overview.md), [docs/design/multiprocessing.md](../../vllm/docs/design/multiprocessing.md) | Process architecture | +| [docs/design/plugin_system.md](../../vllm/docs/design/plugin_system.md), [docs/contributing/model/registration.md](../../vllm/docs/contributing/model/registration.md), [docs/design/custom_op.md](../../vllm/docs/design/custom_op.md), [docs/design/vllm_ir.md](../../vllm/docs/design/vllm_ir.md) | Extension mechanisms | +| [docs/design/torch_compile.md](../../vllm/docs/design/torch_compile.md), [docs/design/cuda_graphs.md](../../vllm/docs/design/cuda_graphs.md), [docs/design/cuda_graphs_multimodal.md](../../vllm/docs/design/cuda_graphs_multimodal.md), [docs/design/optimization_levels.md](../../vllm/docs/design/optimization_levels.md), [docs/design/model_runner_v2.md](../../vllm/docs/design/model_runner_v2.md), [docs/design/dbo.md](../../vllm/docs/design/dbo.md) | Execution design docs | +| [docs/design/prefix_caching.md](../../vllm/docs/design/prefix_caching.md), [docs/features/disagg_encoder.md](../../vllm/docs/features/disagg_encoder.md), [docs/features/prompt_embeds.md](../../vllm/docs/features/prompt_embeds.md) | Caching / feature docs | +| [docs/getting_started/installation/README.md](../../vllm/docs/getting_started/installation/README.md), [docs/getting_started/installation/cpu.md](../../vllm/docs/getting_started/installation/cpu.md), [docs/getting_started/installation/cpu.arm.inc.md](../../vllm/docs/getting_started/installation/cpu.arm.inc.md), [docs/getting_started/installation/cpu.apple.inc.md](../../vllm/docs/getting_started/installation/cpu.apple.inc.md), [docs/getting_started/installation/cpu.x86.inc.md](../../vllm/docs/getting_started/installation/cpu.x86.inc.md), [docs/getting_started/installation/gpu.apple.inc.md](../../vllm/docs/getting_started/installation/gpu.apple.inc.md), [docs/getting_started/installation/gpu.cuda.inc.md](../../vllm/docs/getting_started/installation/gpu.cuda.inc.md) | Hardware/installation docs | +| [docs/features/quantization/README.md](../../vllm/docs/features/quantization/README.md), [docs/features/quantization/gguf.md](../../vllm/docs/features/quantization/gguf.md), [docs/features/quantization/bnb.md](../../vllm/docs/features/quantization/bnb.md) | Quantization support matrix and plugin notes | +| [docs/models/supported_models.md](../../vllm/docs/models/supported_models.md), [docs/models/hardware_supported_models/cpu.md](../../vllm/docs/models/hardware_supported_models/cpu.md) | Model lists (audio, transcription, realtime, OpenVLA; CPU-validated list) | +| [docs/serving/online_serving/openai_compatible_server.md](../../vllm/docs/serving/online_serving/openai_compatible_server.md), [docs/serving/online_serving/speech_to_text.md](../../vllm/docs/serving/online_serving/speech_to_text.md), [docs/contributing/model/transcription.md](../../vllm/docs/contributing/model/transcription.md), [docs/features/multimodal_inputs.md](../../vllm/docs/features/multimodal_inputs.md) | API and audio docs | +| [docs/usage/troubleshooting.md](../../vllm/docs/usage/troubleshooting.md) | Jetson Thor `sm_110a` ptxas note | +| [docs/contributing/model/tests.md](../../vllm/docs/contributing/model/tests.md) | Model test guidance | +| [tests/v1/core/test_scheduler.py](../../vllm/tests/v1/core/test_scheduler.py), [tests/v1/core/test_prefix_caching.py](../../vllm/tests/v1/core/test_prefix_caching.py), [tests/v1/engine/test_engine_core.py](../../vllm/tests/v1/engine/test_engine_core.py), [tests/v1/streaming_input/test_async_llm_streaming.py](../../vllm/tests/v1/streaming_input/test_async_llm_streaming.py), [tests/kernels/test_onednn.py](../../vllm/tests/kernels/test_onednn.py), [tests/platforms/test_cpu.py](../../vllm/tests/platforms/test_cpu.py), [tests/models/multimodal/processing/test_openvla.py](../../vllm/tests/models/multimodal/processing/test_openvla.py), [tests/plugins/vllm_add_dummy_platform/setup.py](../../vllm/tests/plugins/vllm_add_dummy_platform/setup.py), [tests/plugins_tests/test_endpoint_plugins.py](../../vllm/tests/plugins_tests/test_endpoint_plugins.py) | Representative tests | +| [.buildkite/test-pipeline.yaml](../../vllm/.buildkite/test-pipeline.yaml), [.buildkite/hardware_tests/cpu.yaml](../../vllm/.buildkite/hardware_tests/cpu.yaml), [.buildkite/hardware_tests/gh200.yaml](../../vllm/.buildkite/hardware_tests/gh200.yaml), [.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh](../../vllm/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh), [.buildkite/check-wheel-size.py](../../vllm/.buildkite/check-wheel-size.py), [.github/workflows/macos-smoke-test.yml](../../vllm/.github/workflows/macos-smoke-test.yml) | CI | +| [examples/basic/offline_inference/basic.py](../../vllm/examples/basic/offline_inference/basic.py), [examples/generate/multimodal/audio_language_offline.py](../../vllm/examples/generate/multimodal/audio_language_offline.py), [examples/generate/multimodal/vision_language_offline.py](../../vllm/examples/generate/multimodal/vision_language_offline.py), [examples/generate/multimodal/encoder_decoder_multimodal_offline.py](../../vllm/examples/generate/multimodal/encoder_decoder_multimodal_offline.py), [examples/generate/multimodal/qwen2_5_omni/only_thinker.py](../../vllm/examples/generate/multimodal/qwen2_5_omni/only_thinker.py), [examples/generate/multimodal/qwen2_5_omni/README.md](../../vllm/examples/generate/multimodal/qwen2_5_omni/README.md), [examples/speech_to_text/openai/openai_transcription_client.py](../../vllm/examples/speech_to_text/openai/openai_transcription_client.py), [examples/speech_to_text/realtime/openai_realtime_microphone_client.py](../../vllm/examples/speech_to_text/realtime/openai_realtime_microphone_client.py) | Examples | diff --git a/docs/engines/vllm_omni.md b/docs/engines/vllm_omni.md new file mode 100644 index 000000000..294832a0c --- /dev/null +++ b/docs/engines/vllm_omni.md @@ -0,0 +1,575 @@ +# vLLM-Omni (vllm-project/vllm-omni) — Engine Analysis + +| Field | Value | +| --- | --- | +| Local path | `/data/zhoutaichang/embedding_infer/vllm-omni` (links below are relative to `analysis/engines/`, pointing into the sibling `vllm-omni/` checkout) | +| Remote | `https://github.com/vllm-project/vllm-omni.git` (origin) | +| Branch | `main` | +| HEAD | `7be014bce6374f06c95b703763bdbac4c6198f31` — 2026-09-07 "[Bugfix][NPU] Fix MiniMax-H3 INT8 quantization dispatch (#6876)" | +| Dirty status | clean (`git status --short` empty) | +| Submodules | none (`git submodule status` empty) | +| Analysis date | 2026-09-08 | +| Installed package caveat | The Python environment has vllm-omni installed **editable from a different checkout**: `/home/zhoutaichang/feature/vllm-omni-main` @ `be335a86f` (2026-08-28), version string `0.28.0rc2.dev22+gbe335a86f`, with `vllm==0.28.0`. This report analyses the newer `7be014bc` tree only; nothing was executed. [Verified] | +| Size | 1,382 `.py` files / 508,455 LOC under `vllm_omni/`; 912 `test_*.py` files; 266 files under `docs/`; 90 deploy YAMLs. [Verified] | + +Evidence labels: **[Verified]** read in code/docs; **[Inferred]** reasoned from structure; **[Proposal]** design suggestion; **[Unknown]** not determinable from the tree. + +## 0. Summary + +- vLLM-Omni is a **Python-only serving framework layered on top of vLLM (v1 engine)**; it does not ship its own kernels, allocator, or C++ runtime. Its `setup.py` builds a pure-Python wheel and only selects a platform-specific `requirements/*.txt` (`cuda|rocm|npu|xpu|musa|cpu`). [Verified] (`../../vllm-omni/setup.py`, `../../vllm-omni/requirements/common.txt`) +- **vLLM is not pinned in packaging metadata.** No `vllm==` appears in `requirements/` or `pyproject.toml`; instead `version.py` warns at import if major.minor differ, and docs pin `vllm==0.28.0` for the 0.28.x line. [Verified] (`../../vllm-omni/vllm_omni/version.py`, `../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md`) +- It **imports 284 distinct `vllm.*` modules** and 104 files import `vllm.v1.*` internals; `patch.py` **monkeypatches** vLLM classes at import (`EngineCoreRequest/Output(s)`, `Request`, `StreamingUpdate`, `TokensPrompt`, `MRotaryEmbedding`, `ModelConfig.is_mm_prefix_lm`, `CuMemAllocator`, NVFP4/FP8 kernels, torch inductor). Backend substitution below vLLM is therefore not a plugin exercise; it is a re-implementation of the AR runtime. [Verified] (`../../vllm-omni/vllm_omni/patch.py`) +- Core abstraction: a model is a **frozen `PipelineConfig` of stages** (`LLM_AR`, `LLM_GENERATION`, `DIFFUSION`), and a **`DeployConfig` YAML** places stages on devices, sizes them, and wires named connectors. 60 pipelines are registered in `OMNI_PIPELINES`. [Verified] (`../../vllm-omni/vllm_omni/config/stage_config.py`, `../../vllm-omni/vllm_omni/config/pipeline_registry.py`) +- Each AR/generation stage is a **separate vLLM `EngineCoreProc` subprocess** (spawned by `StageEngineCoreProcManager`) reached over ZMQ; diffusion stages run either in a separate `StageDiffusionProc` (ZMQ PUSH/PULL) or inline in the orchestrator process. An `Orchestrator` asyncio loop routes outputs stage→stage. [Verified] +- **Inter-stage tensors travel host-side by default** ("D2H2D") through `OmniConnector`s: `SharedMemoryConnector` (POSIX shm + msgpack); Mooncake (TCP/RDMA) and Yuanrong (Ascend) for multi-node, Mori (RDMA, plus an intra-node AMD `xgmi` backend configured in `deploy/qwen3_omni_moe_mori_intranode.yaml`); `MooncakeTransferEngineConnector.put` and the Mori connector pass `torch.Tensor`/`ManagedBuffer` payloads without msgpack serialization and can use device memory pools (repo docs still describe all connectors as D2H2D); none of these device paths is validated for an edge target. [Verified] (`../../vllm-omni/vllm_omni/distributed/omni_connectors/`) +- **Streaming audio** is produced by `async_chunk`: talker decode steps accumulate codec frames (`codec_chunk_frames`, default 25, with a smaller dynamic first chunk) and push them to the code2wav stage, which decodes chunk-by-chunk while the request sits in a patched `RequestStatus.WAITING_FOR_CHUNK`. [Verified] (`../../vllm-omni/docs/design/feature/async_chunk.md`) +- **Platforms actually in tree:** CUDA, ROCm, Ascend NPU (via `vllm_ascend`), Intel XPU, Moore Threads MUSA (via `vllm_musa`). There is **no CPU platform/worker**; `requirements/cpu.txt` exists only for docs builds and the fallback `UnspecifiedOmniPlatform` reports `device_count()==0`. [Verified] (`../../vllm-omni/vllm_omni/platforms/__init__.py`) +- **No edge/mobile targets**: no Jetson/ARM/Metal/CoreML/Vulkan/OpenCL/QNN/NNAPI code, no ONNX/TensorRT export path (onnxruntime is used only to load a few speaker-embedding / codec helper models). [Verified] +- **TTS is a first-class citizen**: 15 TTS model families, an OpenAI `/v1/audio/speech` endpoint with a per-model `TTSModelAdapter` registry, WebSocket streaming `/v1/audio/speech/stream`, PCM/WAV/MP3/Opus encoding via `soundfile`, and a repo-published Qwen3-TTS TTFP of 64 ms (H200, conc. 1; repo-reported, not reproduced). [Verified] +- **VLA is present but server-side**: π0, GR00T-N1.7, InternVLA-A1 (single diffusion stage returning `actions`), DreamZero-DROID and LingBot-World (AR-diffusion world models with engine-level paged KV), Cosmos3 action modes; served over `WS /v1/realtime/robot/openpi` (msgpack). Actions are not streamed; one observation → one action chunk. [Verified] +- Diffusion has its own runtime (`DiffusionEngine`, `RequestScheduler`/`StepScheduler`, `DiffusionWorker`, 11 attention backends, cache-DiT/TeaCache/MagCache/StepCache, CPU/layerwise/distributed offload, HSDP/Ulysses/Ring/CFG/VAE parallel) that reuses vLLM layers/quant/`KVCacheManager` as a library. [Verified] +- Compilation: AR stages inherit vLLM CUDA graphs/`torch.compile`; diffusion uses `regionally_compile()` (`torch.compile` per repeated block) and per-model CUDA-graph wrappers (e.g., `CUDAGraphDecoderWrapper` for Qwen3-TTS code2wav). [Verified] +- Quantization is inherited from vLLM (`quantization_config` path) with omni additions (Int8, MXFP4/MXFP8 for NPU, SVDQuant, quack FP8, bitsandbytes); non-AR TTS stages stay BF16 per docs. [Verified] (`../../vllm-omni/docs/user_guide/quantization/overview.md`) +- Bottom line for the edge study: vLLM-Omni is the reference **serving stack** (pipeline/stage/streaming semantics, model ports, API contracts) but its **execution runtime is vLLM+CUDA-class accelerators**; an edge port must replace the AR runtime (vLLM EngineCore/scheduler/worker) and the diffusion worker while keeping `PipelineConfig`, stage input processors, connector payload schemas, and the OpenAI/OpenPI protocol layers. [Inferred] + +## 1. Architecture overview + +```mermaid +graph TD + subgraph Entry["Entrypoints (vllm_omni/entrypoints)"] + CLI["vllm serve --omni
cli/serve.py"] + API["OpenAI-compatible FastAPI
openai/api_server.py"] + OFF["Omni / AsyncOmni
omni.py / async_omni.py"] + DUP["Duplex + Realtime WS
duplex/*, openai/realtime_connection.py"] + OPI["OpenPI robot WS
openpi/serving.py"] + end + subgraph Ctrl["Control plane (vllm_omni/engine)"] + ENG["AsyncOmniEngine
async_omni_engine.py"] + ORCH["Orchestrator
orchestrator.py"] + RT["StageRuntime / DistStageRuntime
stage_runtime.py"] + POOL["StagePool (per logical stage)
stage_pool.py"] + MS["OmniMasterServer (ZMQ ROUTER)
stage_engine_startup.py"] + end + subgraph Cfg["Config (vllm_omni/config)"] + PC["PipelineConfig (frozen topology)
stage_config.py"] + DC["DeployConfig YAML
vllm_omni/deploy/*.yaml"] + REG["OMNI_PIPELINES
pipeline_registry.py"] + end + subgraph AR["AR / generation stage (subprocess)"] + ECP["StageEngineCoreProc (vLLM EngineCoreProc)"] + SCH["OmniARScheduler / OmniGenerationScheduler
core/sched/"] + WRK["GPUARWorker / GPUGenerationWorker
worker/"] + MR["GPUARModelRunner / GPUGenerationModelRunner"] + MODEL["Omni model classes
model_executor/models/*"] + end + subgraph DIFF["Diffusion stage (subprocess or inline)"] + SDP["StageDiffusionProc (ZMQ)"] + DE["DiffusionEngine + RequestScheduler/StepScheduler"] + DW["DiffusionWorker / DiffusionModelRunner"] + PIPE["Pipelines
diffusion/models/*"] + end + CONN["OmniConnector (SHM / Mooncake / Mori / Yuanrong)
distributed/omni_connectors"] + PLAT["OmniPlatform (cuda/rocm/npu/xpu/musa)
platforms/"] + VLLM["vLLM 0.28.x (patched by vllm_omni/patch.py)"] + + CLI --> API --> ENG + OFF --> ENG + DUP --> ENG + OPI --> ENG + ENG --> ORCH + ENG --> RT + RT --> POOL + RT --> MS + REG --> PC --> ENG + DC --> ENG + POOL -->|ZMQ| ECP + POOL -->|ZMQ / inline| SDP + ECP --> SCH --> WRK --> MR --> MODEL + SDP --> DE --> DW --> PIPE + MR -->|put/get chunks, KV| CONN + DW -->|put/get| CONN + WRK --> PLAT + DW --> PLAT + ECP --> VLLM + MR --> VLLM +``` + +Key modules: + +| Module | Path | Responsibility | Key symbols | +| --- | --- | --- | --- | +| Package init / patches | `../../vllm-omni/vllm_omni/__init__.py`, `../../vllm-omni/vllm_omni/patch.py` | Version check, then monkeypatch vLLM internals at import | `warn_if_misaligned_vllm_version`, `_patched_is_mm_prefix_lm`, `_patch_cumem_free_callback_cuda` | +| Engine args | `../../vllm-omni/vllm_omni/engine/arg_utils.py` | `OmniEngineArgs(EngineArgs)` per-stage fields; `OrchestratorArgs`; registers omni models into vLLM via entry point | `OmniEngineArgs`, `OrchestratorArgs`, `register_omni_models_to_vllm` | +| Composition root | `../../vllm-omni/vllm_omni/engine/async_omni_engine.py` | Resolves stage configs, boots stages, hosts orchestrator loop, request submit/abort/RPC/sleep | `AsyncOmniEngine`, `_create_default_diffusion_stage_cfg`, `add_request_async`, `abort_async` | +| Orchestrator | `../../vllm-omni/vllm_omni/engine/orchestrator.py` | Cross-stage routing, output ordering, cancellation, CFG companions, PD pair | `Orchestrator`, `_route_output`, `_forward_to_next_stage_unguarded`, `_prewarm_async_chunk_stages`, `_handle_abort` | +| Stage runtime | `../../vllm-omni/vllm_omni/engine/stage_runtime.py`, `../../vllm-omni/vllm_omni/engine/stage_pool.py` | Expand logical stages into local/remote replicas; replica selection/affinity; metrics | `StageRuntime`, `DistStageRuntime`, `StagePool.pick`, `submit_initial` | +| Stage process launch | `../../vllm-omni/vllm_omni/engine/stage_engine_startup.py`, `../../vllm-omni/vllm_omni/engine/stage_engine_core_proc.py`, `../../vllm-omni/vllm_omni/engine/stage_engine_core_proc_manager.py` | ZMQ address allocation, headless replica registration, spawn of `EngineCoreProc` subclasses | `OmniMasterServer`, `StageEngineCoreProc.run_stage_core`, `launch_stage_replica` | +| Stage client | `../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py` | vLLM `AsyncMPClient` subclass per replica; KV sender endpoint | `StageEngineCoreClient`, `DPLBStageEngineCoreClient` | +| Pipeline/deploy config | `../../vllm-omni/vllm_omni/config/stage_config.py`, `../../vllm-omni/vllm_omni/config/pipeline_registry.py`, `../../vllm-omni/vllm_omni/config/omni_config.py` | Frozen topology, deploy overlay, precedence, scheduler resolution | `PipelineConfig`, `StagePipelineConfig`, `StageDeployConfig`, `StageExecutionType`, `_resolve_scheduler`, `OMNI_PIPELINES`, `register_pipeline` | +| AR scheduler | `../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py`, `../../vllm-omni/vllm_omni/core/sched/omni_scheduler_mixin.py` | vLLM `Scheduler` subclass adding chunk/KV-transfer hooks | `OmniARScheduler`, `OmniARAsyncScheduler`, `OmniSchedulerMixin` | +| Generation scheduler | `../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py` | One-shot non-AR stage scheduling (code2wav, DiT-in-vLLM) | `OmniGenerationScheduler.schedule` | +| Workers/runners | `../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py`, `../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py`, `../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py`, `../../vllm-omni/vllm_omni/worker/omni_connector_model_runner_mixin.py` | vLLM `GPUWorker`/`GPUModelRunner` subclasses; multimodal outputs, talker MTP, connector I/O | `GPUARWorker`, `GPUARModelRunner.execute_model`, `GPUGenerationModelRunner`, `OmniConnectorModelRunnerMixin.send_chunk` | +| Model registry | `../../vllm-omni/vllm_omni/model_executor/models/registry.py`, `../../vllm-omni/vllm_omni/diffusion/registry.py` | 89 omni AR/generation archs + all vLLM archs; 66 diffusion pipelines | `OmniModelRegistry`, `DiffusionModelRegistry`, `register_diffusion_model` | +| Stage input processors | `../../vllm-omni/vllm_omni/model_executor/stage_input_processors/` | Model-owned functions converting stage-N output → stage-N+1 input (full payload or async chunk) | `talker2code2wav_async_chunk` (qwen3_tts.py) | +| Diffusion engine | `../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py`, `../../vllm-omni/vllm_omni/diffusion/sched/`, `../../vllm-omni/vllm_omni/diffusion/executor/`, `../../vllm-omni/vllm_omni/diffusion/worker/` | Request/step batching, uniproc/multiproc executors, worker & runner | `DiffusionEngine`, `RequestScheduler`, `StepScheduler`, `MultiprocDiffusionExecutor`, `DiffusionWorker`, `DiffusionModelRunner` | +| Diffusion attention | `../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py`, `../../vllm-omni/vllm_omni/diffusion/attention/selector.py`, `../../vllm-omni/vllm_omni/diffusion/attention/layer.py` | Enum registry, per-role selection, platform defaults | `DiffusionAttentionBackendEnum`, `register_diffusion_backend`, `get_attn_backend_for_role`, `Attention` | +| Connectors | `../../vllm-omni/vllm_omni/distributed/omni_connectors/` | Transport-only put/get + chunk transfer adapter | `OmniConnectorBase`, `SharedMemoryConnector`, `OmniConnectorFactory`, `OmniChunkTransferAdapter` | +| Platforms | `../../vllm-omni/vllm_omni/platforms/interface.py`, `../../vllm-omni/vllm_omni/platforms/__init__.py` | Backend detection, worker class selection, diffusion attention defaults | `OmniPlatform`, `resolve_current_omni_platform_cls_qualname`, `CudaOmniPlatform`, `NPUOmniPlatform` | +| I/O types | `../../vllm-omni/vllm_omni/inputs/data.py`, `../../vllm-omni/vllm_omni/outputs/__init__.py`, `../../vllm-omni/vllm_omni/data_entry_keys.py`, `../../vllm-omni/vllm_omni/request.py` | Prompt types with embeds/payloads; `OmniRequestOutput`; typed inter-stage payload | `OmniTokensPrompt`, `OmniRequestOutput`, `OmniPayload`, `OmniRequest` | +| Serving | `../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py`, `../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py`, `../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py` | HTTP/WS routes; TTS request building & audio encoding; robot policy WS | `omni_run_server`, `create_speech`, `ServingRealtimeRobotOpenPI.infer` | +| Experimental | `../../vllm-omni/vllm_omni/experimental/ar_diffusion/engine.py`, `../../vllm-omni/vllm_omni/experimental/world_models/` | AR-diffusion (world model) engine with paged KV; session state | `ARDiffusionEngine`, `ARDiffusionModelRunner` | + +## 2. Device layer + +**Backends and registration.** `vllm_omni/platforms/__init__.py` defines five builtin probe functions and an entry-point group `vllm_omni.platform_plugins` for out-of-tree platforms; exactly one may activate. [Verified] (`../../vllm-omni/vllm_omni/platforms/__init__.py` — `builtin_omni_platform_plugins`, `resolve_current_omni_platform_cls_qualname`) + +| Platform | Probe | Class (inherits vLLM/vendor platform) | AR/generation worker | Notes | +| --- | --- | --- | --- | --- | +| CUDA | `pynvml.nvmlDeviceGetCount()>0` | `CudaOmniPlatform(OmniPlatform, CudaPlatformBase)` `../../vllm-omni/vllm_omni/platforms/cuda/platform.py` | `GPUARWorker`, `GPUGenerationWorker` | Default deploy dir `vllm_omni/deploy`; diffusion attention default chain (TRTLLM→cuDNN→FlashInfer→FA→SDPA) keyed on SM version [Verified] | +| ROCm | `amdsmi` handles>0 | `RocmOmniPlatform(OmniPlatform, RocmPlatform)` `../../vllm-omni/vllm_omni/platforms/rocm/platform.py` | same GPU workers | Forces `TRITON_ATTN`/`ROCM_AITER_FA` for AR; `patch_groupnorm.py`; several deploy YAMLs flip code2wav to eager (MIOpen not capture-safe) [Verified] | +| Ascend NPU | `torch.npu.is_available()` | `NPUOmniPlatform(OmniPlatform, vllm_ascend.NPUPlatform)` `../../vllm-omni/vllm_omni/platforms/npu/platform.py` | `NPUARWorker`, `NPUGenerationWorker` (`../../vllm-omni/vllm_omni/platforms/npu/worker/`) | HCCL; applies model patches for code2wav conv ops, 310P patches; `YuanrongTransferEngineConnector`; ACL graph wrapper [Verified] | +| Intel XPU | `torch.xpu.is_available()` | `XPUOmniPlatform(OmniPlatform, XPUPlatform)` `../../vllm-omni/vllm_omni/platforms/xpu/platform.py` | `XPUARWorker`, `XPUGenerationWorker` | Disables XPU sampler kernel via env [Verified] | +| MUSA | `torchada.is_musa_platform()` | `MUSAOmniPlatform(OmniPlatform, vllm_musa.MUSAPlatformBase)` `../../vllm-omni/vllm_omni/platforms/musa/platform.py` | GPU workers | requires `torchada`, `mate`, `flash_attn_3` (`../../vllm-omni/requirements/musa.txt`) [Verified] | +| CPU | — | `UnspecifiedOmniPlatform` (`device_type="cpu"`, `get_device_count()==0`) | none | No CPU worker/runner exists; `cli/serve.py` falls back to vLLM `CpuPlatform` **only for argument parsing** [Verified] | + +**Device discovery/selection.** Placement is declarative: each stage's `devices: "0,1"` in the deploy YAML must equal TP×DP_local×PP (×replicas); `StageRuntime._scoped_spawn_device_env` sets the platform's `device_control_env_var` (e.g., `CUDA_VISIBLE_DEVICES`) around subprocess spawn, and `GPUARWorker.init_device` maps local rank → visible device via `current_omni_platform.get_torch_device`. [Verified] (`../../vllm-omni/vllm_omni/engine/stage_runtime.py`, `../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py`, `../../vllm-omni/docs/configuration/stage_configs.md`) + +**Memory allocation/pools.** +- AR/generation stages inherit vLLM's paged KV allocation; `OmniGPUWorkerBase.determine_available_memory` re-implements the budget as `total*gpu_memory_utilization − (weights + peak activation + non-torch)` measured by `memory_profiling` around `profile_run()`, or honors explicit `kv_cache_memory_bytes`. [Verified] (`../../vllm-omni/vllm_omni/worker/base.py`) +- Multi-stage co-location on one GPU is guarded by `stage_admission.py` (`DeviceLedger`, `check_admission`) using per-stage `gpu_memory_utilization` plus a CUDA-graph reserve, and by SH/EX device locks for sequential profiling (`parallel_stage_init`). [Verified] (`../../vllm-omni/vllm_omni/engine/stage_admission.py`) +- Sleep mode uses vLLM's `CuMemAllocator` tagged pools (`_maybe_get_memory_pool_context`), patched in `patch.py` to avoid a CUDA double-free on shutdown. [Verified] +- Diffusion workers: `DiffusionWorker.init_device` calls `init_distributed_environment`/`initialize_model_parallel` from `vllm_omni.diffusion.distributed`, and only takes a `MemorySnapshot`/KV budget when `diffusion_kv_mode=paged_scheduler`. Host-side: pinned memory for offload (`pin_cpu_memory`), CUDA host registration for DLO, and the Host Weight Runtime (mmap'd safetensors shared across same-host workers). [Verified] (`../../vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py`, `../../vllm-omni/vllm_omni/diffusion/offloader/`, `../../vllm-omni/docs/design/feature/host_weight_runtime.md`) +- NVML process-scoped accounting helpers remain for diffusion memory reporting. [Verified] (`../../vllm-omni/vllm_omni/worker/gpu_memory_utils.py`) + +**Host↔device transfers.** Inter-stage data is always staged to host: connector `put()` serializes CPU tensors (msgpack, `OmniMsgpackEncoder`, "zero-copy not implemented yet") into a POSIX shm segment; the receiving stage deserializes and re-uploads. `GPUARModelRunner` snapshots multimodal outputs to CPU on a dedicated copy stream (`_get_or_create_omni_payload_copy_stream`, `_snapshot_tensor_payload_to_cpu_async`) so D2H does not block decode ("async omni output materialization"). [Verified] (`../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py`, `../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py`, `../../vllm-omni/docs/design/feature/omni_async_output_materialization.md`) + +**Synchronization/streams.** `OmniPlatform.record_device_event()` marks tensor readiness for async diffusion output (NPU overrides to sync the default stream first, since HCCL uses hidden streams). Prefix cache writes record a CUDA event on the copy stream (`_PendingAsyncWrite`). Ruff bans `torch.cuda.*` in favor of `torch.accelerator.*` to keep code device-neutral. [Verified] (`../../vllm-omni/vllm_omni/platforms/interface.py`, `../../vllm-omni/vllm_omni/core/prefix_cache.py`, `../../vllm-omni/pyproject.toml`) + +**Portability.** Linux only (`docs/getting_started/quickstart.md`: "OS: Linux"); Python 3.10–3.13; Dockerfiles for cuda/rocm/xpu/npu (A2/A3). No macOS/Windows/ARM CPU/mobile code paths were found. [Verified] (`../../vllm-omni/docker/Dockerfile.cuda`, `../../vllm-omni/docker/Dockerfile.npu`) + +## 3. Kernel layer + +**Operator set & registration.** vLLM-Omni has no compiled extension of its own; operators come from (a) vLLM's layers (`vllm.model_executor.layers.{linear,layernorm,activation,rotary_embedding,fused_moe,quantization}` — 62/25/15/9/5/17 import sites respectively) and vLLM's custom-op/IR mechanism, (b) PyTorch/Triton code inside model files (e.g., `attention/fish_kvcache_triton.py`, `diffusion/layers/mot/ops/mot_gemm.py`, `diffusion/layers/fused_qk_norm_rope.py`), and (c) optional external kernel packages (flash-attn 2/3/4, FlashInfer, SageAttention 2/3, cuDNN SDPA, HF `kernels` hub, quack FP8, TRT-LLM gen). [Verified] + +**Dispatch mechanisms.** +- AR stages: attention/backend dispatch is vLLM's (`attention_backend` engine arg, forced to `TRITON_ATTN`/`ROCM_AITER_FA` on ROCm by `stage_init_utils.py`). Omni adds a Fish-Speech-specific KV-cache attention Triton path selected by model arch (`is_fish_kvcache_attn_active_for_model`) with fallback counters. [Verified] (`../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py`) +- Diffusion attention: `Attention` layer → `get_attn_backend_for_role(role, head_size, attention_config)` with precedence per-role config → role category → global default → `current_omni_platform.get_diffusion_attn_backend_cls()`; classes resolved from `DiffusionAttentionBackendEnum` (11 members) with runtime overrides via `register_diffusion_backend`. Each backend declares `supported_platforms`, `validate_available()`, head-size sets, mask/paged-KV capabilities. [Verified] (`../../vllm-omni/vllm_omni/diffusion/attention/selector.py`, `../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py`, `../../vllm-omni/vllm_omni/diffusion/attention/backends/abstract.py`) +- Diffusion generic ops: `CustomOp.dispatch_forward()` picks `forward_cuda|hip|npu|xpu|musa|native` from `current_omni_platform`; HIP/MUSA default to the CUDA path, XPU to native. [Verified] (`../../vllm-omni/vllm_omni/diffusion/layers/custom_op.py`) +- IR op priority: CUDA platform prefers `["vllm_c","native"]` (optionally `oink` for rms_norm); per-pipeline `get_*_ir_op_priority_func` (Cosmos3) can merge overrides. [Verified] (`../../vllm-omni/vllm_omni/platforms/cuda/platform.py`, `../../vllm-omni/vllm_omni/diffusion/registry.py`) + +**Quantization formats & where dequant happens.** Unified `quantization_config` path (`../../vllm-omni/vllm_omni/quantization/factory.py`); vLLM quant methods are reused (ModelOpt FP8/NVFP4, AutoRound W4A16, TorchAO, compressed-tensors) and omni adds `int8_config.py`, `mxfp4_config.py`, `mxfp8_config.py` (Ascend), `svdquant_config.py`, `bitsandbytes_config.py`, `inc_config.py`, `quack_fp8.py` (Blackwell fused-bias FP8 GEMM). Dequant/scaled-GEMM occurs inside vLLM's `LinearMethod.apply`/`ScaledMMLinearKernel` (patched in `patch.py` for non-contiguous batched diffusion activations and FlashInfer output shape). GGUF loading for diffusion via `gguf_adapters` (`requirements/common.txt`). Runtime FP8 attention quant for diffusion FA (`AttnQuantSpec`). Docs state omni/TTS quantization is scoped to the AR LM stage; non-AR stages stay BF16. [Verified] (`../../vllm-omni/docs/user_guide/quantization/overview.md`, `../../vllm-omni/vllm_omni/quantization/`) + +**Compilation / JIT / graph capture.** +- AR stages: vLLM `compilation_config` (`cudagraph_mode: PIECEWISE|FULL`, `custom_ops`) is settable per stage in deploy YAML (see `platforms.npu.stages[*].compilation_config` in `../../vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml`). `GPUARModelRunner._capture_talker_mtp_graphs` captures a dedicated CUDA graph for the talker MTP/code-predictor path; `supports_talker_mtp_graph_capture()` is a platform hook. NPU uses `ACLGraphWrapper` via `get_graph_wrapper_cls()`. [Verified] +- Generation stages (code2wav): model-owned wrappers, e.g., `CUDAGraphDecoderWrapper` with capture sizes derived from chunk ramp/left context (`decode_cudagraph_capture_sizes`, `decode_cudagraph_batch_sizes` connector extras). [Verified] (`../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py`, `../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml`) +- Diffusion: `regionally_compile()` applies `torch.compile` to each repeated block (`_repeated_blocks`), keeping offload hook wrappers outside the graph; `diffusion_compile_granularity="regional"`, `diffusion_compile_dynamic=True` defaults; `enforce_eager` disables. `patch.py` also patches torch inductor's `statically_known_multiple_of` for FLUX fp8 dynamic shapes. [Verified] (`../../vllm-omni/vllm_omni/diffusion/compile.py`, `../../vllm-omni/docs/user_guide/diffusion/regional_compilation.md`) + +**Fallback when an op is unsupported.** Diffusion attention: explicit selection of an unavailable backend raises at platform resolution (fail-fast); automatic selection degrades to `TORCH_SDPA`. `CustomOp.forward_native` is the last resort. Fish KV attention records fallback reasons and uses vLLM attention otherwise. Platform-specific model patches (NPU code2wav conv2d, ROCm groupnorm) replace individual ops. [Verified] + +**Extension points.** +- Out-of-tree platform: entry point `vllm_omni.platform_plugins` returning a class path; general plugins `vllm_omni.general_plugins`. [Verified] (`../../vllm-omni/vllm_omni/plugins/__init__.py`) +- Attention backend: `register_diffusion_backend(enum, "pkg.Class")`. [Verified] +- Model: `register_diffusion_model(arch, module, cls, pre/post-process funcs)`; AR models via `OmniModelRegistry` (vLLM `_ModelRegistry`); pipelines via `register_pipeline`. [Verified] +- Connector: `OmniConnectorFactory.register_connector(name, ctor)`. [Verified] (`../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py`) +- Custom diffusion pipeline without registry edits: `diffusion_load_format="dummy"` + `custom_pipeline_args={"pipeline_class": ...}` + `CustomPipelineWorkerExtension`. [Verified] (`../../vllm-omni/docs/features/custom_pipeline.md`) +- TTS serving adapter: `@register_tts_adapter` subclasses of `TTSModelAdapter` (`ARTTSAdapter`, `DiffusionTTSAdapter`). [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/__init__.py`, `../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/base.py`) +- Contributor guides: `../../vllm-omni/docs/contributing/model/adding_omni_model.md`, `../../vllm-omni/docs/contributing/model/adding_tts_model.md`, `../../vllm-omni/docs/contributing/model/adding_diffusion_model.md`. [Verified] + +## 4. Model runner + +**Model loading / formats.** +- AR/generation stages: standard vLLM loader path (HF `config.json` + safetensors; `trust_remote_code`); the omni arch is looked up in `OmniModelRegistry` (vLLM archs + `_OMNI_MODELS`), and `register_omni_models_to_vllm` is installed as a `vllm.general_plugins` entry point so vLLM worker subprocesses that only import `vllm` still see omni archs. Per-stage `hf_config_name`, `model_subdir`, `tokenizer_subdir`, `model_path_resolver` select sub-configs inside multi-component repos. [Verified] (`../../vllm-omni/pyproject.toml`, `../../vllm-omni/vllm_omni/config/stage_config.py`) +- Diffusion stages: `DiffusersPipelineLoader` reads Diffusers-style `model_index.json` repos (safetensors index or unindexed shards, GGUF via `download_gguf`, ModelOpt/direct-mmap checkpoint adapters, multithreaded weight load, HWR host-weight plan); `diffusion_load_format ∈ {default, dummy, diffusers}` where `diffusers` runs the stock `diffusers` pipeline through `DiffusersAdapterPipeline`. Some VLA pipelines self-load (π0 loads `model.safetensors` in `__init__`; GR00T uses `AutoModel`/`AutoProcessor`). [Verified] (`../../vllm-omni/vllm_omni/diffusion/model_loader/diffusers_loader.py`, `../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py`, `../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py`) +- No export/conversion pipeline exists (no ONNX/TensorRT/GGUF-export tooling; GGUF is consumed, not produced). [Verified] +- Pipeline auto-detection: `StageConfigFactory.create_from_model()` maps HF `model_type`/`architectures`/`_class_name` to `OMNI_PIPELINES`; single-stage diffusion models without a registered pipeline use `_create_default_diffusion_stage_cfg`. [Verified] (`../../vllm-omni/vllm_omni/config/config_factory.py`, `../../vllm-omni/vllm_omni/engine/async_omni_engine.py`) + +**Execution lifecycle (init → warmup → step).** +1. `AsyncOmniEngine.__init__` → `_resolve_stage_configs` → `StageRuntime.initialize()` builds `LogicalStageInitPlan`/`ReplicaInitPlan`, runs admission, spawns replicas (LLM via `launch_stage_replica` → `StageEngineCoreProcManager` → `StageEngineCoreProc.run_stage_core`; diffusion via `StageDiffusionProcManager.launch_headless` or `InlineStageDiffusionClient`). [Verified] +2. Inside an LLM stage, vLLM's `EngineCore` init runs `init_device` → `load_model` → `determine_available_memory` (profile run) → `initialize_kv_cache` → `compile_or_warm_up_model` (CUDA graph capture); omni overrides `load_model` (`_init_talker_mtp`, `_prewarm_attention_capture_workspaces`) and `capture_model` (`_capture_talker_mtp_graphs`). [Verified] (`../../vllm-omni/vllm_omni/worker/gpu_model_runner.py`, `../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py`) +3. Diffusion: `DiffusionEngine.make_engine()` → executor init → `load_model` → `run_startup_warmup()` (`_dummy_run`, KV profile requests when paged) → `_busy_loop` thread performing `schedule → execute_batch/execute_step → update_from_output → per-request asyncio queue`. [Verified] (`../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py`, `../../vllm-omni/docs/design/feature/diffusion_continuous_batching.md`) +4. Orchestrator `_prewarm_async_chunk_stages` pre-submits downstream stage requests when `async_chunk` is on, so they can park in `WAITING_FOR_CHUNK`. [Verified] +5. Step: AR stage `GPUARModelRunner.execute_model` → `_preprocess` (model-specific `preprocess` builds `inputs_embeds`; `_build_model_kwargs_extra`) → `_model_forward` → `sample_tokens` (may run `_run_post_sample_talker_mtp`) → `_build_omni_step_outputs` → `OmniModelRunnerOutput` with `multimodal_outputs`/`pooler_output`. [Verified] + +**Dynamic shapes.** AR stages rely on vLLM's padded batch buckets (`max_cudagraph_capture_size`, `cudagraph_mode`); talker MTP graphs are captured per batch bucket. Code2wav graph shapes are derived from the chunk-size schedule (`initial_codec_chunk_frames`, `codec_chunk_frames`, `codec_left_context_frames`, optional `codec_chunk_ramp`/adaptive ramp) and batch buckets; eager fallback on unmatched shapes (`_get_padded_size` returns None). Diffusion uses `diffusion_compile_dynamic=True` and per-request `DiffusionRequestBatch` slicing. [Verified] + +**State & KV cache.** +- AR/generation: inherits vLLM paged KV (`KVCacheManager`, block tables, prefix caching, chunked prefill, preemption). Omni additions: `OmniTensorPrefixCache` stores hidden states and multimodal tensors aligned to vLLM block slot mappings so prefix hits reuse stage outputs (`../../vllm-omni/vllm_omni/core/prefix_cache.py`, `../../vllm-omni/docs/design/feature/prefix_caching.md`); `retains_state_across_chunks` keeps a request resident while awaiting chunks; KV transfer between stages (`omni_kv_config`, `send_kv_cache`/`recv_kv_cache`, rank-aware keys) for PD disaggregation and thinker→talker reuse. [Verified] +- Across calls (sessions): streaming input sessions (`OmniStreamingUpdate`, `resumable` requests, `session_id`) and duplex sessions retain KV between segments; `session_mode: duplex` in deploy YAML. [Verified] (`../../vllm-omni/vllm_omni/request.py`, `../../vllm-omni/docs/design/fullduplex.md`) +- Diffusion: `DiffusionKVCacheMode ∈ {dense_legacy, paged_scheduler}`; `DiffusionKVCacheManager` wraps vLLM `KVCacheManager` (caching disabled) for scheduler-owned paged KV in AR-DiT/world models ("unified AR/DiT paged KV cache runtime", 0.28.0). Experimental `ARDiffusionEngine` keeps runner-owned paged KV per session with sliding window + sink frames (`ARDiffusionKVCacheSpec`). Prompt-embedding cache (`enable_prompt_embed_cache`). Eviction: LRU-style session accounting in `experimental/world_models/session_state`. [Verified] (`../../vllm-omni/vllm_omni/diffusion/diffusion_kv/manager.py`, `../../vllm-omni/vllm_omni/experimental/ar_diffusion/capability.py`) + +**Multimodal stages.** +- Encoders (audio/vision) live inside the thinker model class (vLLM multimodal processors, `mm_features`); `requires_multimodal_data` lets a downstream stage receive raw media. [Verified] +- Projectors/talkers: `LLM_AR` stage with `engine_output_type="latent"` emitting codec codes via `multimodal_outputs`. [Verified] +- Decoders/vocoders: `LLM_GENERATION` stage (`GPUGenerationModelRunner`: no logits/sampling, returns tensors via `pooler_output`) e.g. `Qwen3TTSCode2Wav`, `Qwen3OmniMoeCode2Wav`, `MossTTSCodecDecoder`, `FishSpeechDACDecoder`, `IndexTTS2S2MelDecoder`; or a DIFFUSION stage for flow-matching decoders (`CosyVoice3` DiT, `MiniMaxMusic3Acoustic`, `OmniVoicePipeline`). [Verified] (`../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py`, `../../vllm-omni/vllm_omni/model_executor/models/registry.py`) +- Diffusion (image/video/audio/action) as DIFFUSION stage pipelines. [Verified] + +## 5. Scheduler + +### (a) Request / token / pipeline-level scheduling + +- **Per-stage continuous batching** is inherited from vLLM: each LLM stage runs its own `Scheduler` subclass chosen by `_resolve_scheduler(execution_type, async_scheduling)`: `OmniARScheduler` (sync), `OmniARAsyncScheduler` (vLLM `AsyncScheduler`), `OmniGenerationScheduler`; diffusion stages return `None` (own scheduler). Batching knobs are per stage (`max_num_seqs`, `max_num_batched_tokens`, `enable_chunked_prefill`, `enable_prefix_caching`, `async_scheduling`). [Verified] (`../../vllm-omni/vllm_omni/config/stage_config.py`) +- **AR schedule hooks**: `OmniARScheduler.schedule` drops aborted requests, `_process_pending_omni_inputs`, optionally defers waiting admission, calls `super().schedule()`, restores omni wait queues, post-processes with cached payloads, and wraps output with `finished_requests_needing_kv_transfer`. `update_from_output` triggers `chunk_transfer_adapter.save_async` per decode step and handles KV-transfer readiness. [Verified] (`../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py`) +- **Generation schedule** (`OmniGenerationScheduler.schedule`): one-shot fast path feeding all input tokens at once (placeholder token if 0), chunk-aware: skips placeholder tokens when no new chunk arrived, finishes when `is_done_receiving_chunks`. [Verified] +- **Chunk lifecycle** (`OmniChunkTransferAdapter`): keys `{req_id}_{stage_id}_{chunk_id}`; background `recv_loop`/`save_loop` threads; `process_pending_chunks` parks requests in `WAITING_FOR_CHUNK` (enum value −1 added by `patch.py`), `restore_queues` re-admits when ready, `_preempt_non_active_running` with `active_stream_window`. Full-payload (non-async) handoff uses `WAITING_FOR_INPUT` (−2) and `OmniSchedulingCoordinator`. [Verified] (`../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py`, `../../vllm-omni/vllm_omni/core/sched/omni_scheduling_coordinator.py`) +- **Pipeline-level**: `Orchestrator._orchestration_loop` (poll) or `_orchestration_loop_event_driven` (`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`) reads each replica's `EngineCoreOutputs`, runs `MultimodalOutputProcessor`, then `_route_output`: final-output stages → `OutputMessage` to client queue; non-final → `_forward_to_next_stage` (unless the next stage receives async chunks, in which case data flows via connectors and the stage was pre-submitted). Diffusion next stages get inputs via `custom_process_input_func(source_outputs, prompt, requires_multimodal_data)`; CFG companions are bundled (`CfgCompanionTracker`); PD pairs forward original prompt + `kv_transfer_params`. [Verified] (`../../vllm-omni/vllm_omni/engine/orchestrator.py`, `../../vllm-omni/vllm_omni/engine/cfg_companion_tracker.py`) +- **Replica selection / load balancing**: `StagePool.pick`/`select_replica_id` with `LoadBalancer` policies (`--omni-lb-policy`, default `random`), request→replica binding for affinity; `OmniCoordinator` tracks heartbeats for distributed replicas. [Verified] (`../../vllm-omni/vllm_omni/engine/stage_pool.py`, `../../vllm-omni/vllm_omni/distributed/omni_coordinator/omni_coordinator.py`) +- **Diffusion scheduling**: `REQUEST_BATCH` (`RequestScheduler`: fuse compatible requests by `RequestBatchSamplingParamsKey`, optional `request_batch_max_wait_ms` admission window) or `STEP_BATCH` (`StepScheduler`: continuous batching at denoise-step granularity, `max_num_seqs>1`), plus `streaming_output` for chunked video and `submit_interaction` for mid-stream prompt events. [Verified] (`../../vllm-omni/vllm_omni/diffusion/sched/request_scheduler.py`, `../../vllm-omni/vllm_omni/diffusion/sched/step_scheduler.py`, `../../vllm-omni/docs/user_guide/diffusion/execution_modes.md`) +- **Cancellation/abort**: `AsyncOmni.abort` → `AsyncOmniEngine.abort_async` → `AbortRequestMessage` → `Orchestrator._handle_abort` → `StagePool.abort_requests` on every stage holding the request; AR final-stage abort yields a terminal output with `finish_reason="abort"` and the generated prefix; diffusion abort is whole-sample (`DiffusionEngine.abort` → `_process_aborts_queue`); connector `cleanup(request_id)` unlinks shm segments; `abort_requests_collecting_outputs` in the output processor; `FINISHED_ABORTED` requests are dropped before vLLM sees them. [Verified] (`../../vllm-omni/docs/design/module/engine_orchestration.md`) +- **Backpressure**: generation admission gate in `AsyncOmni._submit_with_admission`; `duplex_max_sessions` engine-owned session capacity; `max_num_seqs` per stage; `collect_timed_out_request_ids` for chunk/input timeouts (`connector_get_max_wait*`); `active_stream_window` limits concurrently fed downstream streams. [Verified] + +### (b) Graph / operator / thread-level scheduling + +- Within a stage step the model forward is a single torch call sequence (eager, piecewise or full CUDA graph) — there is **no omni-level operator DAG scheduler**; ordering is the model's Python. [Verified] +- Thread pools: chunk transfer adapter `recv_loop`/`save_loop` threads; async omni output builder thread (`OmniAsyncGPUModelRunnerOutput._build_output_in_background`); `DiffusionEngine._busy_loop` thread; multiproc diffusion `_result_pump` thread; `orchestrator_monitor`. [Verified] +- Stream assignment: default compute stream plus a dedicated payload copy stream for D2H snapshots and prefix-cache writes; NPU synchronizes default stream before events. Diffusion sequence-parallel comms use `GroupCoordinator` in `vllm_omni.diffusion.distributed`. [Verified] +- Diffusion regional compilation is the only compile-graph scheduling; N/A otherwise. [Verified] + +## 6. I/O layer + +- **Preprocessing.** Media parsing reuses `vllm.multimodal` (audio via `vllm.multimodal.audio` — `librosa` is banned by ruff in favor of vLLM helpers/torchaudio); `utils/audio.py` provides mel filterbanks; per-model processors under `transformers_utils/processors`; TTS reference audio encoding happens in serving adapters (`_encode_moss_references`, `_build_qwen3_tts_request`); diffusion image preprocessing via `get_*_pre_process_func`. [Verified] (`../../vllm-omni/vllm_omni/utils/audio.py`, `../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py`) +- **Tokenization.** HF `transformers` tokenizers through vLLM's tokenizer layer (`vllm.tokenizers` imported in 20 files); `owns_tokenizer` marks the stage whose tokenizer the pipeline exposes; custom `MammothModa2Tokenizer`; speech tokenizers/codecs per model (e.g., `Qwen3TTSTokenizer` 12 Hz/25 Hz, `s3tokenizer`, `whisper_vq`). [Verified] (`../../vllm-omni/vllm_omni/tokenizers/mammoth_moda2_tokenizer.py`, `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_tokenizer.py`) +- **Streaming.** Text deltas via vLLM `RequestOutputKind.DELTA`; audio via async chunk `OmniEngineCoreOutput.multimodal_output` per chunk, accumulated by `OmniRequestState.add_multimodal_tensor` / `multimodal_accumulation.py`; HTTP TTS streams PCM through `_generate_pcm_chunks`/`_iter_pcm_audio_bytes` or SSE (`_generate_audio_sse_events`); WebSocket `OmniStreamingSpeechHandler` (`/v1/audio/speech/stream`, optional word alignments via forced aligner); `StreamingAudioResampler` for integer-ratio downsampling (24 kHz→8 kHz). Video streaming output via `/v1/video/chat/stream`, `/v1/realtime/video`. [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py`, `../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py`, `../../vllm-omni/vllm_omni/outputs/output_processor.py`) +- **Buffering.** Talker→code2wav frame accumulation (`code_prompt_token_ids` deque per request; chunk = 25 frames, left context 25–72 frames, first chunk 1–4 or dynamic); shm segments per chunk; per-request asyncio queues for diffusion; duplex `realtime_input.py` audio buffers with Silero VAD (`SileroStreamingVAD`, 16 kHz). [Verified] (`../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py`, `../../vllm-omni/vllm_omni/entrypoints/duplex/vad.py`) +- **Postprocessing.** Detokenize via vLLM output processor (`sampling_constraints={"detokenize": False}` on talkers); vocoders as generation/diffusion stages; diffusion `get_*_post_process_func` converts latents → PIL/video/actions; audio encoding with `soundfile` (`wav`, `pcm`, `mp3`, `opus`), speed change via torchaudio phase-vocoder. [Verified] +- **Application boundaries.** + - Python: `Omni.generate(prompts, sampling_params_list, py_generator=...)`, `AsyncOmni` (implements vLLM `EngineClient`). [Verified] (`../../vllm-omni/vllm_omni/entrypoints/omni.py`, `../../vllm-omni/vllm_omni/entrypoints/async_omni.py`) + - CLI: `vllm serve --omni` (vLLM subcommand plugin) and `vllm-omni` console script; stage-based launch `--stage-id N --headless --omni-master-address/-port`; `vllm-omni bench serve`. [Verified] (`../../vllm-omni/vllm_omni/entrypoints/cli/main.py`, `../../vllm-omni/vllm_omni/entrypoints/cli/serve.py`) + - HTTP (FastAPI): `/v1/chat/completions` (+`/batch`), `/v1/audio/speech` (+`/batch`), `/v1/audio/generate`, `/v1/audio/voices` (GET/POST/DELETE), `/v1/images/generations`, `/v1/images/edits`, `/v1/videos` (+`/sync`, list/get/delete/content), `/v1/models`, `/health`, `/v1/omni/sleep|wakeup`, `/start_profile|/stop_profile`. WebSocket: `/v1/audio/speech/stream`, `/v1/video/chat/stream`, `/v1/realtime/video`, `/v1/realtime` (turn-based PCM), `/v1/realtime/robot/openpi`, `/v1/duplex`. [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py`) + - No C API, no mobile bindings, no gRPC. [Verified] + +## 7. Execution flow + +### (a) Text LLM decode (thinker-only pipeline, e.g. `qwen3_omni_moe_thinker_only`) + +```mermaid +sequenceDiagram + participant C as Client (HTTP) + participant S as api_server / OmniOpenAIServingChat + participant A as AsyncOmni.generate + participant E as AsyncOmniEngine + participant O as Orchestrator (asyncio) + participant P as StagePool[0] / StageEngineCoreClient (ZMQ) + participant EC as StageEngineCoreProc (vLLM EngineCore) + participant SC as OmniARAsyncScheduler + participant W as GPUARWorker / GPUARModelRunner + C->>S: POST /v1/chat/completions (stream) + S->>A: generate(prompt, sampling_params_list) + A->>E: add_request_async (StageSubmissionMessage) + E->>O: request queue + O->>P: submit_initial(req) -> OmniEngineCoreRequest + P->>EC: ZMQ add_request + loop each engine step + EC->>SC: schedule() + SC-->>EC: SchedulerOutput (prefill/decode batch) + EC->>W: execute_model -> sample_tokens + W-->>EC: OmniModelRunnerOutput (token ids, hidden states opt.) + EC->>SC: update_from_output + EC-->>P: OmniEngineCoreOutputs (ZMQ) + P-->>O: poll_llm_raw_output -> MultimodalOutputProcessor + O-->>E: OutputMessage(final_output=True, DELTA) + E-->>A: _process_orchestrator_results + A-->>S: OmniRequestOutput + S-->>C: SSE chunk + end + C-->>S: disconnect / cancel + S->>A: abort(request_id) + A->>O: AbortRequestMessage + O->>P: abort_requests -> EC finish_requests(FINISHED_ABORTED) +``` +[Verified] for the participants and call names (`../../vllm-omni/vllm_omni/entrypoints/async_omni.py`, `../../vllm-omni/vllm_omni/engine/orchestrator.py`, `../../vllm-omni/vllm_omni/engine/stage_pool.py`); exact vLLM-internal ZMQ framing inherited from `AsyncMPClient` [Inferred]. + +### (b) TTS with async chunk streaming (Qwen3-TTS: talker → code2wav) + +```mermaid +sequenceDiagram + participant C as Client + participant SP as OmniOpenAIServingSpeech (+Qwen3TTS adapter) + participant O as Orchestrator + participant T as Stage0 talker (OmniARScheduler + GPUARModelRunner) + participant AD as OmniChunkTransferAdapter (stage0 sched proc) + participant SHM as SharedMemoryConnector (/dev/shm) + participant G as Stage1 code2wav (OmniGenerationScheduler + GPUGenerationModelRunner) + participant M as Qwen3TTSCode2Wav (+CUDAGraphDecoderWrapper) + C->>SP: POST /v1/audio/speech {input, voice, stream} + SP->>SP: build prompt (prompt_embeds/additional_information, ref audio codes) + SP->>O: AsyncOmni.generate (2 sampling params) + O->>T: submit stage0 request + O->>G: _prewarm_async_chunk_stages: pre-submit stage1 request (WAITING_FOR_CHUNK) + loop each talker decode step + T->>T: execute_model -> codec frame in multimodal_outputs + T->>AD: update_from_output -> save_async(pooler_output, request) + AD->>AD: talker2code2wav_async_chunk: accumulate frames; emit when >= initial/steady chunk (1..25 frames + left ctx) + AD->>SHM: put(key=req_stage_chunk, OmniPayloadStruct{codes,meta}) + end + G->>SHM: recv_loop get(key) -> chunk ready + G->>G: restore_queues -> schedule (append chunk tokens) + G->>M: execute_model -> chunked_decode_with_cudagraph + M-->>G: waveform chunk (per request via ubatch_slices) + G-->>O: OmniEngineCoreOutput.multimodal_output {audio} + O-->>SP: OutputMessage(stage1 final_output=True) + SP->>SP: soundfile encode (pcm/wav/mp3/opus), optional resample + SP-->>C: chunked HTTP / SSE / WS audio + T-->>AD: finished -> meta.finished chunk + G-->>O: last chunk, finished=True -> cleanup shm segments +``` +[Verified] (`../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py`, `../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/shm_connector.py`, `../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py`, `../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml`, `../../vllm-omni/docs/design/feature/async_chunk.md`). + +VLA path (π0 / GR00T): `WS /v1/realtime/robot/openpi` → `ServingRealtimeRobotOpenPI.infer(obs, session_id, reset)` → `_build_request` puts the raw observation into `sampling_params.extra_args["robot_obs"]` → `AsyncOmni.generate` → single DIFFUSION stage (`DiffusionEngine.step_streaming`) → `Pi0Pipeline.forward(req)` (preprocess + flow-matching denoise) → `DiffusionOutput(output={"actions": ndarray})` → `multimodal_output["actions"]` → msgpack reply. [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py`, `../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py`) + +## 8. Tests and examples + +- **Layout**: `../../vllm-omni/tests/` (912 test files) split by module: `engine/` (48), `core/` (21), `worker/` (14), `diffusion/` (311), `entrypoints/` (66), `e2e/` (180: `offline_inference/`, `online_serving/`, `accuracy/`, `features/`), `model_executor/` (136), `distributed/` (18), `platforms/` (9), `examples/` (17), plus `dfx/` (perf/reliability/stability). Root `conftest.py` is thin; fixtures in `tests/helpers/fixtures/`. [Verified] (`../../vllm-omni/tests/conftest.py`) +- **Markers / levels** (pyproject): `core_model` (L1/L2 per PR), `advanced_model` (L3 per merge), `full_model` (L4 nightly), `local_model`; module markers `diffusion|omni|tts|cache|parallel|sp|example`; platform markers `cpu|gpu|cuda|rocm|xpu|npu|musa`; SKU markers `H100|H800|H200|L4|B200|MI325|B60|S5000|A2|A3`; `cards_1..8`. [Verified] (`../../vllm-omni/pyproject.toml`) +- **How to run** (`../../vllm-omni/docs/contributing/ci/test_execution_guide.md`): `uv pip install ".[dev]"; apt-get install espeak-ng jq`; L1: `cd tests && pytest -s -v -m "core_model and cpu"`; L2/L3/L4 via `bash tools/run_ready_jobs.sh|run_merge_jobs.sh|tools/nightly/run_nightly_jobs.sh` which read `.buildkite/cuda/test-{ready,merge,nightly}.yml`; ad hoc `pytest -s -v test_x.py --run-level=core_model`. [Verified] +- **CI**: Buildkite pipelines for cuda (`../../vllm-omni/.buildkite/cuda/test-ready.yml`, `test-merge.yml`, `test-nightly.yml`, `test-weekly.yml`), amd, intel (XPU, `VLLM_VERSION: v0.28.0`), npu (A2/A3); GitHub Actions only `pre-commit.yml` and `build_wheel.yml`. Nightly docker `vllm/vllm-omni:nightly`. [Verified] +- **Relevant examples**: TTS `../../vllm-omni/examples/offline_inference/text_to_speech/` (qwen3_tts, cosyvoice3, fish_speech, gepard, glm_tts, higgs_audio_v2/v3, indextts2, ming_tts, moss_tts(_nano), omnivoice, voxcpm2, voxtral_tts; `qwen3_tts/word_timestamps.py`) and `../../vllm-omni/examples/online_serving/text_to_speech/`; omni `../../vllm-omni/examples/offline_inference/qwen3_omni/end2end_async_chunk.py`, `../../vllm-omni/examples/online_serving/qwen3_omni/openai_realtime_client.py`; VLA `../../vllm-omni/examples/online_serving/pi0/openpi_client.py`, `../../vllm-omni/examples/online_serving/dreamzero/openpi_client.py`, `../../vllm-omni/examples/offline_inference/internvla_a1/end2end.py`, `../../vllm-omni/examples/offline_inference/dreamzero/client_schedule.py`; custom pipeline `../../vllm-omni/examples/offline_inference/custom_pipeline/image_to_image/custom_pipeline.py`; apps `../../vllm-omni/apps/ComfyUI-vLLM-Omni`. No mobile examples. [Verified] +- **Relevant e2e tests**: `../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_base.py`, `test_qwen3_omni_colocate_async.py`, `test_lingbot_world_v2_stepwise.py`; `../../vllm-omni/tests/e2e/online_serving/test_pi0_expansion.py`, `test_gr00t_openpi_expansion.py`, `test_dreamzero_expansion.py`, `test_minicpmo_4_5_duplex.py`. [Verified] +- **Benchmarks**: `../../vllm-omni/benchmarks/tts/bench_tts.py` (+`model_configs.yaml`), `benchmarks/diffusion`, `benchmarks/kernels`, `vllm-omni bench serve` (TTFP/RTF metrics in `vllm_omni/metrics`). [Verified] + +## 9. TTS relevance + +**Pipeline-level facts.** +- TTS models are two-stage `LLM_AR` (talker → codec tokens) + `LLM_GENERATION` (code2wav) or + `DIFFUSION` (flow-matching acoustic decoder) pipelines; single-stage native-AR TTS also exists (Gepard). `sampling_constraints={"detokenize": False, "stop_token_ids": [...]}` keep vLLM from detokenizing codec ids. [Verified] (`../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/pipeline.py`) +- Streaming audio is produced by `async_chunk: true` with connector extras (`codec_streaming`, `codec_chunk_frames`, `codec_left_context_frames`, `initial_codec_chunk_frames`, `codec_chunk_ramp`, `codec_chunk_adaptive`, `decode_batch_max_size`, `decode_cudagraph_*`). Dynamic initial chunk sizing (`compute_dynamic_initial_chunk_size`) trades TTFP vs load. [Verified] (`../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml`, `../../vllm-omni/vllm_omni/model_executor/stage_input_processors/chunk_size_utils.py`) +- Code2wav batching across requests uses `ubatch_slices` to split batched waveforms; CUDA graphs captured for the stateful streaming decoder shapes. [Verified] (`../../vllm-omni/docs/design/feature/async_chunk.md`, `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py`) +- Serving: `/v1/audio/speech` with per-model adapters (`qwen3_tts`, `cosyvoice3`, `fish_speech`, `voxtral`, `moss_tts`, `higgs_audio_v2/v3`, `indextts2`, `ming_tts`, `ming_flash_omni_tts`, `glm_tts`, `dots_tts`, `voxcpm2`, `omnivoice`, `step_audio2`, `covo_audio`, `audex`, `audex_tta`, `minimax_music3`); voices registry endpoints (`/v1/audio/voices`, `custom_voice_dir`), speaker cache (`utils/speaker_cache.py`), forced aligner for word timestamps (`utils/forced_aligner.py`, `--forced-aligner`), speech usage accounting (`speech_usage.py`). [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/`) +- Realtime speech-to-speech: `/v1/realtime` (turn-based) and full-duplex `/v1/duplex` / `/v1/realtime?duplex=1` with server VAD (Silero), used by MiniCPM-o 4.5, PersonaPlex (Moshi finetune), Nemotron VoiceChat. [Verified] (`../../vllm-omni/docs/serving/realtime_duplex_api.md`, `../../vllm-omni/docs/serving/full_duplex_api.md`) +- Repo-reported performance (not reproduced): Qwen3-TTS on H200, concurrency 1: E2E 941 ms, TTFP 64 ms, RTF 0.16 vs HF 15,513 ms / 2.64; Qwen3-Omni on A100: TTFP 0.934 s, RTF 0.32. [Verified as repo claims] (`../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md`) + +**TTS/audio model families in tree** (registry keys in `../../vllm-omni/vllm_omni/model_executor/models/registry.py`, dirs under `../../vllm-omni/vllm_omni/model_executor/models/`): + +| Family | Dir | Stages | Vocoder/codec | +| --- | --- | --- | --- | +| Qwen3-TTS (CustomVoice/VoiceDesign/Base) | `qwen3_tts` | Talker (AR) → `Qwen3TTSCode2Wav` (gen) | 12 Hz/25 Hz speech tokenizer, ConvNet code2wav, x-vector speaker enc | +| CosyVoice3 | `cosyvoice3` (+ `diffusion/models/cosyvoice3_audio`) | Talker → flow-matching DiT code2wav | s3tokenizer; ONNX speaker embedding | +| Fish Speech S2 Pro | `fish_speech` | Slow AR (+fast AR) → `FishSpeechDACDecoder` | DAC; custom KV-cache Triton attention | +| Gepard-1.0 | `gepard` | single-stage native AR | FSQ/NanoCodec | +| VoxCPM2 | `voxcpm2` | Talker (+ diffusion head) | — | +| dots.tts | `dots_tts` | Talker | — | +| Voxtral TTS | `voxtral_tts` | AR → `VoxtralTTSAudioGeneration` (flow matching) + audio tokenizer | — | +| MOSS-TTS family (Delay/Realtime/Local, TTSD, SoundEffect, VoiceGenerator) | `moss_tts`, `moss_tts_nano` | Talker → `MossTTSCodecDecoder` | MOSS codec | +| IndexTTS-2 / 2.5 | `indextts2` | Talker → `IndexTTS2S2MelDecoder` | s2mel + vocoder | +| GLM-TTS | `glm_tts` | Talker (+ voice clone helper, ONNX) | — | +| Higgs-Audio v2 / v3 | `higgs_audio_v2`, `higgs_audio_v3` | Talker → Code2Wav | — | +| OmniVoice | `omnivoice` (+ `diffusion/models/omnivoice`) | diffusion pipeline | — | +| Ming-TTS (dense/MoE), Ming-flash-omni TTS | `ming_tts`, `ming_flash_omni` | LLM → `MingAudioVAEModel` | audio VAE | +| Audex (Nemotron) TTS/TTA/S2S | `audex` | Thinker → `AudexCode2Wav` / `AudexXCodec1` | XCodec | +| Step-Audio2, MiMo-Audio, Covo-Audio, MiniCPM-o 4.5, Qwen2.5/3-Omni, Nemotron VoiceChat, PersonaPlex, Dynin-Omni, Aura-Omni | respective dirs | omni thinker → talker → token2wav | model codecs (Token2Wav / CosyVoice2 flow) | +| MiniMax-Music3 (text-to-music) | `minimax_music3` | Talker → acoustic flow decoder | — | +| Stable-Audio-Open | `diffusion/models/stable_audio` | diffusion | — | + +**What's missing for edge TTS.** No CPU or mobile execution path for talker or vocoder; vocoder stages assume CUDA graphs or Ascend/ROCm patches; audio chunks cross a POSIX-shm + msgpack boundary (fine on a server, wasteful on a single-process edge device); no low-latency in-process single-stage mode (a stage is always a vLLM `EngineCore` subprocess with its own scheduler); no quantization validated for talker/code2wav stages (docs: "not validated"); no INT8/INT4 vocoder path; `Omni` offline API is batch-oriented (`py_generator=True` streams finished requests, not audio chunks — chunk streaming is only via `AsyncOmni`/HTTP). [Verified/Inferred] + +## 10. VLA relevance + +**Pipeline-level facts.** +- VLA policies are **single `DIFFUSION` stages** with `final_output_type="action"|"actions"`: `PI0_PIPELINE` (`../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py`), `GR00T_N1D7_PIPELINE` (`../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py`), InternVLA-A1 (`InternVLAA1Pipeline`, default single-stage fallback). Observations arrive in `sampling_params.extra_args["robot_obs"]`; the pipeline owns all preprocessing (image resize, tokenization of the language instruction, state normalization) and returns `DiffusionOutput(output={"actions": ...})`. [Verified] +- Vision encoders: π0 uses PaliGemma (SigLIP + Gemma-2B prefix, `paligemma_variant`, `action_expert_variant: gemma_300m`, 224×224, up to 3 cameras); GR00T N1.7 uses its own VLM backbone through `Gr00tN1d7`/`Gr00tN1d7Processor` with embodiment-conditioned MLP and DiT action head (`modeling/modules/dit.py`, `embodiment_conditioned_mlp.py`); InternVLA-A1 uses a Qwen3-VL adapter (`adapter_qwen3_vl.py`) with a Cosmos-based world/action model (`model_cosmos.py`). [Verified] (`../../vllm-omni/vllm_omni/deploy/pi0.yaml`, `../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py`, `../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py`) +- Structured/proprioceptive inputs: π0 `max_state_dim: 32`, `max_action_dim: 32`, `chunk_size: 50`, `num_inference_steps: 10`; GR00T `dataio/state_action/{state_action_processor,action_chunking,pose}.py` and `embodiment_tags.py` (DROID, G1, R1 Pro, LIBERO, SimplerEnv …); DreamZero `transform/droid.py`, `roboarena.py`, `action_encoder.py`. [Verified] +- Action heads / diffusion-flow policies: flow matching (π0 `sample_actions` loop, Euler; GR00T DiT; InternVLA Cosmos CI); Cosmos3 action modes `policy|forward_dynamics|inverse_dynamics` with embodiment→domain id map (`../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py`). [Verified] +- World models: DreamZero-DROID (`DreamZeroPipeline`, `engine_backend: ARDiffusionEngine`, `step_cache` DiT cache, `dreamzero_tp1_cfg2.yaml` CFG-parallel) and LingBot-World 2.0 (`LingBotWorldCausalDMDPipeline`, stepwise deploy `lingbot_world_v2_stepwise.yaml`) run on the experimental AR-diffusion engine: per-session paged KV (`ARDiffusionKVCacheSpec`: tokens_per_frame, window/sink frames), tick protocol with `session_id/event_id/chunk_index/request_id`, one latent block per request. SANA-WM (`sana_wm`) and Cosmos3 world generation also exist. [Verified] (`../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md`, `../../vllm-omni/vllm_omni/deploy/dreamzero.yaml`) +- Low-latency loop: OpenPI WebSocket (`/v1/realtime/robot/openpi`, msgpack + numpy ext, 64 MiB max frame, 30 s idle close, `session_id`/`reset` semantics, first frame = `policy_server_config`); request→response per observation; `max_num_seqs: 1`, `enforce_eager: true`, float32 for π0 by default. Session state manager (`enable_session_state_manager`, `experimental/world_models/session_state`) keeps model state across observations. [Verified] (`../../vllm-omni/docs/serving/openpi_api.md`, `../../vllm-omni/docs/features/session_state_manager.md`) +- Deploy YAMLs: `../../vllm-omni/vllm_omni/deploy/pi0.yaml`, `../../vllm-omni/vllm_omni/deploy/Gr00tN1d7.yaml`, `../../vllm-omni/vllm_omni/deploy/dreamzero.yaml`, `../../vllm-omni/vllm_omni/deploy/dreamzero_tp1_cfg2.yaml`, `../../vllm-omni/vllm_omni/deploy/lingbot_world_v2_stepwise.yaml`. [Verified] + +**What's missing for edge VLA.** No action-token streaming (one reply per observation); no explicit control-loop latency budget/deadline scheduling; VLA pipelines are eager PyTorch modules (no CUDA graph / compile path exercised for π0/GR00T by default; GR00T loads via HF `AutoModel`); no CPU/Jetson-specific path (π0 picks `cuda` if available else falls to CPU only implicitly via `torch.cuda.is_available()` in GR00T — untested); KV/state reuse across observations is only implemented for world models (AR-diffusion), not for VLM-prefix caching in π0/GR00T; no quantized VLA recipes. [Verified/Inferred] + +## 11. Edge deployment profile + +- **Binary size / dependencies.** Pure-Python wheel; runtime deps (`../../vllm-omni/requirements/common.txt`): `transformers>=5.10.1,<5.15`, `diffusers==0.40.0`, `kernels==0.15.2`, `accelerate`, `cache-dit`, `torchsde`, `openai-whisper`, `imageio[ffmpeg]`, `x-transformers`, `einops`, `pyzmq`, `janus`, `msgpack`, `pydantic`, `gguf`, `cosmos-guardrail`, `soundfile`, `av`; platform files add `onnxruntime` (+`fa3-fwd` on CUDA, `onnxruntime-cann`+`torchaudio` on NPU, `torchada/mate/flash_attn_3` on MUSA, `auto-round-lib` on XPU). Plus vLLM 0.28.x itself (compiled CUDA kernels) and PyTorch. Docker images derive from `vllm/vllm-openai:v0.28.0`. Footprint is multi-GB; no size-optimized build exists. [Verified] +- **Build system.** `setuptools` + `setuptools-scm`; `setup.py` picks the requirements file from `VLLM_OMNI_TARGET_DEVICE` or torch introspection; version suffix `+rocm|+npu|+xpu|+musa|+cpu`. No CMake, no native code. [Verified] (`../../vllm-omni/setup.py`) +- **Supported platforms** (docs `../../vllm-omni/docs/getting_started/installation/README.md`): NVIDIA CUDA (CC ≥ 7.0 per docs; diffusion FA needs ≥ 8.0; Blackwell needs FA4/cuDNN ≥ 9.5), AMD ROCm, Intel XPU, MThreads MUSA, Ascend NPU (Atlas A2/A3, 310P patches). Recipes mention consumer GPUs (RTX 4090/5090, RTX PRO, DGX Spark GB10 for MiniMax-H3). [Verified] (`../../vllm-omni/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md`) +- **Embedded NVIDIA / Jetson**: no Jetson/aarch64-specific code, docs, or CI; the DGX Spark (GB10, aarch64+Blackwell) recipe is the closest evidence that an aarch64 CUDA host can run it, contingent on a vLLM aarch64 wheel. [Verified: absence] [Inferred: GB10 feasibility] +- **ARM CPU, Apple Silicon/Metal/CoreML, Android GPU/NPU (Vulkan/OpenCL/QNN/NNAPI)**: none. `grep` for `metal|coreml|vulkan|opencl|qnn|nnapi|jetson` over `vllm_omni/` and `docs/` found no implementation. [Verified: absence] +- **Model conversion path & constraints.** Models are consumed directly as HF/Diffusers checkpoints (safetensors; GGUF for some diffusion models; ModelOpt/AutoRound/TorchAO pre-quantized checkpoints). There is no exporter to ONNX/TensorRT/CoreML/TFLite; the only "conversion" is loader-side (weight renaming, TP slicing, online quantization, Host Weight Runtime artifacts). Constraints: vLLM's model-class contract (`forward(input_ids, positions, intermediate_tensors, inputs_embeds)`), vLLM paged attention layout, and `OmniPayload` schemas between stages. [Verified] +- **Process model on the edge.** Minimum footprint for a 2-stage TTS is: API server process + orchestrator thread + 2 vLLM `EngineCoreProc` subprocesses (each with scheduler + worker) + shm connector; single-stage diffusion can run inline (`inline_diffusion`/`uni` executor). [Verified] (`../../vllm-omni/docs/configuration/stage_configs.md`) +- [Proposal] For an edge port, retain: `PipelineConfig`/`StagePipelineConfig` topology descriptions, `stage_input_processors` (chunking math in `chunk_size_utils.py`), `OmniPayload` typed schemas, TTS adapters' request building, OpenPI/realtime protocol handlers; replace: `EngineCoreProc` stages with in-process executors, `SharedMemoryConnector` with in-memory queues, vLLM model classes with the target runtime's graphs. + +## 12. Limitations and unknowns + +- [Verified] Hard dependency on vLLM 0.28.x internals (284 modules, v1 engine, monkeypatched classes); minor-version drift is expected to break (`version.py` warning, `patch.py` FRAGILITY notes). +- [Verified] No CPU/edge/mobile execution backend; `requirements/cpu.txt` is for docs builds only. +- [Verified] The default single-node transport is host-staged (D2H2D) even when stages share a GPU (`docs/design/feature/disaggregated_inference.md`); zero-copy is a TODO in `OmniMsgpackEncoder`. Mooncake TE and Mori have device-memory fast paths, including Mori's intra-node XGMI backend; their applicability to an edge target is unvalidated. +- [Verified] Each LLM stage is a separate subprocess with its own memory budget; co-location relies on `gpu_memory_utilization` partitioning and admission checks. +- [Verified] Quantization for TTS/omni non-AR stages is documented as not validated; MXFP4/MXFP8 are Ascend-only. +- [Verified] Many design docs are `status: draft` with "candidate invariants" (engine orchestration, stage runtime, connectors, platforms); the stage-client/process refactor #5441 is in flight. +- [Verified] Experimental subsystems: AR-diffusion engine/world models (`vllm_omni/experimental/ar_diffusion`), full-duplex JoyVL/Mage-VL (`experimental/fullduplex`), `PAGED_WORKER_LOCAL` KV mode reserved-but-rejected. +- [Verified] VLA serving returns whole action chunks; no intermediate streaming; `session_id`/`reset` ignored by π0 (stateless). +- [Inferred] Latency floor for TTS on a single device includes subprocess ZMQ hops, shm serialization, and scheduler polling (`connector_get_sleep_s: 0.01`), which matter more on edge SoCs than on H100. +- [Unknown] Actual runtime behavior of the `7be014bc` tree was not executed here (installed editable is `be335a86f`); no benchmark numbers were reproduced. +- [Unknown] Whether vLLM's aarch64/Jetson wheels satisfy the kernel packages omni's CUDA path probes (`fa3-fwd`, FlashInfer, FA4) — not determinable from this tree. +- [Unknown] Memory footprint of the smallest supported TTS deployment (e.g., Qwen3-TTS 1.7B two-stage) — deploy YAMLs use `gpu_memory_utilization: 0.3` per stage on H100 but no absolute numbers are published in-tree beyond recipes. + +## 13. Reference index + +| Path | Role | +| --- | --- | +| `../../vllm-omni/pyproject.toml` | Package metadata, extras, pytest markers, `vllm.general_plugins` entry point | +| `../../vllm-omni/setup.py` | Platform-aware dependency routing, version suffix | +| `../../vllm-omni/requirements/common.txt` | Common runtime deps (no vllm pin) | +| `../../vllm-omni/requirements/cpu.txt` | CPU requirements (docs builds) | +| `../../vllm-omni/requirements/musa.txt` | MUSA deps | +| `../../vllm-omni/README.md` | Project overview, model classes | +| `../../vllm-omni/docker/Dockerfile.cuda` | CUDA image from `vllm/vllm-openai:v0.28.0` | +| `../../vllm-omni/docker/Dockerfile.npu` | Ascend image | +| `../../vllm-omni/vllm_omni/__init__.py` | Import-time version check + patch | +| `../../vllm-omni/vllm_omni/version.py` | vLLM major.minor alignment warning | +| `../../vllm-omni/vllm_omni/patch.py` | Monkeypatches of vLLM/torch/transformers | +| `../../vllm-omni/vllm_omni/plugins/__init__.py` | Omni plugin groups | +| `../../vllm-omni/vllm_omni/platforms/__init__.py` | Platform probes and resolution | +| `../../vllm-omni/vllm_omni/platforms/interface.py` | `OmniPlatform` abstract hooks, `UnspecifiedOmniPlatform` | +| `../../vllm-omni/vllm_omni/platforms/cuda/platform.py` | CUDA platform, diffusion attention default chain | +| `../../vllm-omni/vllm_omni/platforms/rocm/platform.py` | ROCm platform | +| `../../vllm-omni/vllm_omni/platforms/npu/platform.py` | Ascend platform (vllm_ascend) | +| `../../vllm-omni/vllm_omni/platforms/npu/worker/` | NPU AR/generation workers and runners | +| `../../vllm-omni/vllm_omni/platforms/xpu/platform.py` | Intel XPU platform | +| `../../vllm-omni/vllm_omni/platforms/musa/platform.py` | MUSA platform | +| `../../vllm-omni/vllm_omni/engine/arg_utils.py` | `OmniEngineArgs`, `OrchestratorArgs`, model registration | +| `../../vllm-omni/vllm_omni/engine/async_omni_engine.py` | `AsyncOmniEngine` composition root | +| `../../vllm-omni/vllm_omni/engine/orchestrator.py` | Cross-stage routing, abort, CFG, PD | +| `../../vllm-omni/vllm_omni/engine/cfg_companion_tracker.py` | CFG companion bookkeeping | +| `../../vllm-omni/vllm_omni/engine/stage_runtime.py` | Local/distributed replica lifecycle | +| `../../vllm-omni/vllm_omni/engine/stage_pool.py` | Per-stage replica pool, LB, metrics | +| `../../vllm-omni/vllm_omni/engine/stage_engine_startup.py` | ZMQ master server, replica launch | +| `../../vllm-omni/vllm_omni/engine/stage_engine_core_proc.py` | `StageEngineCoreProc` | +| `../../vllm-omni/vllm_omni/engine/stage_engine_core_proc_manager.py` | Subprocess spawning per replica | +| `../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py` | ZMQ client per replica | +| `../../vllm-omni/vllm_omni/engine/stage_admission.py` | Device memory admission | +| `../../vllm-omni/vllm_omni/engine/__init__.py` | `OmniEngineCoreRequest/Output(s)` | +| `../../vllm-omni/vllm_omni/config/stage_config.py` | `PipelineConfig`, `StageDeployConfig`, scheduler resolution | +| `../../vllm-omni/vllm_omni/config/pipeline_registry.py` | `OMNI_PIPELINES` | +| `../../vllm-omni/vllm_omni/config/config_factory.py` | Pipeline auto-detection from HF config | +| `../../vllm-omni/vllm_omni/config/omni_config.py` | Typed control-plane config | +| `../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml` | Qwen3-TTS deploy (chunk streaming knobs) | +| `../../vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml` | Qwen3-Omni 3-stage deploy with platform overrides | +| `../../vllm-omni/vllm_omni/deploy/pi0.yaml` | π0 deploy | +| `../../vllm-omni/vllm_omni/deploy/Gr00tN1d7.yaml` | GR00T deploy | +| `../../vllm-omni/vllm_omni/deploy/dreamzero.yaml` | DreamZero deploy (AR-diffusion engine) | +| `../../vllm-omni/vllm_omni/deploy/dreamzero_tp1_cfg2.yaml` | DreamZero CFG-parallel deploy | +| `../../vllm-omni/vllm_omni/deploy/lingbot_world_v2_stepwise.yaml` | LingBot-World stepwise deploy | +| `../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py` | AR scheduler | +| `../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py` | Generation scheduler | +| `../../vllm-omni/vllm_omni/core/sched/omni_scheduler_mixin.py` | Shared omni scheduling state | +| `../../vllm-omni/vllm_omni/core/sched/omni_scheduling_coordinator.py` | Full-payload input coordination | +| `../../vllm-omni/vllm_omni/core/prefix_cache.py` | Hidden-state/multimodal prefix cache | +| `../../vllm-omni/vllm_omni/worker/base.py` | Worker base, memory profiling, sleep | +| `../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py` | AR worker | +| `../../vllm-omni/vllm_omni/worker/gpu_model_runner.py` | Shared omni runner (MRoPE, talker MTP init, preprocess) | +| `../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py` | AR runner execute/sample/async output | +| `../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py` | Non-AR generation runner | +| `../../vllm-omni/vllm_omni/worker/omni_connector_model_runner_mixin.py` | Runner-side connector I/O (chunks, payloads, KV) | +| `../../vllm-omni/vllm_omni/worker/gpu_memory_utils.py` | NVML memory helpers | +| `../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py` | Fish Speech Triton KV attention | +| `../../vllm-omni/vllm_omni/model_executor/models/registry.py` | Omni AR/generation model registry (89 archs) | +| `../../vllm-omni/vllm_omni/model_executor/models/output_templates.py` | `OmniOutput` | +| `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/pipeline.py` | Qwen3-TTS topology | +| `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py` | Code2wav CUDA graph capture | +| `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_tokenizer.py` | Qwen3-TTS speech tokenizer | +| `../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py` | GR00T topology | +| `../../vllm-omni/vllm_omni/model_executor/models/dreamzero/pipeline.py` | DreamZero topology | +| `../../vllm-omni/vllm_omni/model_executor/models/lingbot_world/pipeline.py` | LingBot-World topology | +| `../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py` | Talker→code2wav chunk/full-payload processors | +| `../../vllm-omni/vllm_omni/model_executor/stage_input_processors/chunk_size_utils.py` | Chunk ramp / dynamic IC math | +| `../../vllm-omni/vllm_omni/diffusion/registry.py` | Diffusion pipeline registry (66) | +| `../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py` | `DiffusionEngine` | +| `../../vllm-omni/vllm_omni/diffusion/sched/request_scheduler.py` | Request-level batching | +| `../../vllm-omni/vllm_omni/diffusion/sched/step_scheduler.py` | Step-level continuous batching | +| `../../vllm-omni/vllm_omni/diffusion/executor/` | Uniproc/multiproc executors | +| `../../vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py` | Diffusion worker, device init, sleep | +| `../../vllm-omni/vllm_omni/diffusion/worker/request_batch.py` | `DiffusionRequestBatch` | +| `../../vllm-omni/vllm_omni/diffusion/compile.py` | Regional `torch.compile` | +| `../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py` | Attention backend enum/overrides | +| `../../vllm-omni/vllm_omni/diffusion/attention/backends/abstract.py` | Backend capability contract | +| `../../vllm-omni/vllm_omni/diffusion/attention/selector.py` | Per-role backend resolution | +| `../../vllm-omni/vllm_omni/diffusion/attention/layer.py` | Diffusion `Attention` layer | +| `../../vllm-omni/vllm_omni/diffusion/layers/custom_op.py` | Platform op dispatch | +| `../../vllm-omni/vllm_omni/diffusion/diffusion_kv/manager.py` | Scheduler-owned paged KV for diffusion | +| `../../vllm-omni/vllm_omni/diffusion/model_loader/diffusers_loader.py` | Diffusers-style checkpoint loader | +| `../../vllm-omni/vllm_omni/diffusion/offloader/` | CPU/layerwise/distributed offload | +| `../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py` | π0 topology | +| `../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py` | π0 pipeline | +| `../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py` | GR00T pipeline | +| `../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py` | GR00T policy wrapper | +| `../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py` | GR00T model | +| `../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/pipeline_internvla_a1.py` | InternVLA-A1 pipeline | +| `../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py` | InternVLA-A1 policy | +| `../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py` | Cosmos3 action modes | +| `../../vllm-omni/vllm_omni/experimental/ar_diffusion/engine.py` | `ARDiffusionEngine` | +| `../../vllm-omni/vllm_omni/experimental/ar_diffusion/capability.py` | AR-diffusion KV spec contract | +| `../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/base.py` | `OmniConnectorBase` | +| `../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/shm_connector.py` | Shared-memory connector | +| `../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py` | Connector factory/registry | +| `../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py` | Chunk lifecycle adapter | +| `../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py` | msgpack encoder for tensors | +| `../../vllm-omni/vllm_omni/distributed/omni_coordinator/omni_coordinator.py` | Replica membership/heartbeats | +| `../../vllm-omni/vllm_omni/inputs/data.py` | Omni prompt types | +| `../../vllm-omni/vllm_omni/outputs/__init__.py` | `OmniRequestOutput` | +| `../../vllm-omni/vllm_omni/outputs/output_processor.py` | Multimodal output processor | +| `../../vllm-omni/vllm_omni/data_entry_keys.py` | `OmniPayload` schema | +| `../../vllm-omni/vllm_omni/request.py` | `OmniRequest`, `OmniStreamingUpdate` | +| `../../vllm-omni/vllm_omni/entrypoints/omni.py` | Offline `Omni` | +| `../../vllm-omni/vllm_omni/entrypoints/async_omni.py` | `AsyncOmni` EngineClient | +| `../../vllm-omni/vllm_omni/entrypoints/cli/main.py` | CLI interception of `--omni` | +| `../../vllm-omni/vllm_omni/entrypoints/cli/serve.py` | `vllm serve --omni` subcommand | +| `../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py` | HTTP/WS routes | +| `../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py` | TTS serving | +| `../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py` | TTS WebSocket streaming | +| `../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py` | Audio encode/resample | +| `../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/__init__.py` | TTS adapter registry | +| `../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/base.py` | `TTSModelAdapter` | +| `../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py` | OpenPI robot serving | +| `../../vllm-omni/vllm_omni/entrypoints/duplex/vad.py` | Silero VAD | +| `../../vllm-omni/vllm_omni/quantization/factory.py` | Quantization config factory | +| `../../vllm-omni/vllm_omni/quantization/` | Omni quant configs | +| `../../vllm-omni/vllm_omni/utils/audio.py` | Mel filterbank utilities | +| `../../vllm-omni/vllm_omni/tokenizers/mammoth_moda2_tokenizer.py` | Custom tokenizer | +| `../../vllm-omni/docs/design/architecture_overview.md` | Architecture overview (mermaid, layers) | +| `../../vllm-omni/docs/design/module/engine_orchestration.md` | Orchestration contract | +| `../../vllm-omni/docs/design/module/stage_runtime.md` | Stage runtime contract | +| `../../vllm-omni/docs/design/module/omni_connector.md` | Connector contract | +| `../../vllm-omni/docs/design/module/execution_platforms.md` | Platform contract | +| `../../vllm-omni/docs/design/module/ar_runtime.md` | AR runtime contract | +| `../../vllm-omni/docs/design/module/input_output_modality_contracts.md` | I/O contracts | +| `../../vllm-omni/docs/design/feature/async_chunk.md` | Async chunk design + perf table | +| `../../vllm-omni/docs/design/feature/disaggregated_inference.md` | Connectors/D2H2D | +| `../../vllm-omni/docs/design/feature/diffusion_continuous_batching.md` | Diffusion batching modes | +| `../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md` | AR-diffusion sessions | +| `../../vllm-omni/docs/design/feature/host_weight_runtime.md` | Host weight runtime | +| `../../vllm-omni/docs/design/feature/prefix_caching.md` | Omni prefix caching | +| `../../vllm-omni/docs/design/feature/omni_async_output_materialization.md` | Async output materialization | +| `../../vllm-omni/docs/design/fullduplex.md` | Full-duplex runtime | +| `../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md` | Repo-reported TTS perf | +| `../../vllm-omni/docs/configuration/stage_configs.md` | Pipeline/deploy schema, precedence, stage CLI | +| `../../vllm-omni/docs/getting_started/installation/README.md` | Platform list, vLLM alignment | +| `../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md` | `vllm==0.28.0` pin, CC≥7.0 | +| `../../vllm-omni/docs/getting_started/quickstart.md` | Linux/Python 3.12, offline API | +| `../../vllm-omni/docs/user_guide/quantization/overview.md` | Quant modes × hardware matrix | +| `../../vllm-omni/docs/user_guide/diffusion/execution_modes.md` | Diffusion execution modes | +| `../../vllm-omni/docs/user_guide/diffusion/cpu_offload.md` | Offload strategies | +| `../../vllm-omni/docs/user_guide/diffusion/regional_compilation.md` | Regional compile | +| `../../vllm-omni/docs/features/custom_pipeline.md` | Custom pipeline extension | +| `../../vllm-omni/docs/features/session_state_manager.md` | Session state manager | +| `../../vllm-omni/docs/serving/openpi_api.md` | OpenPI WS protocol | +| `../../vllm-omni/docs/serving/speech_api.md` | TTS HTTP API | +| `../../vllm-omni/docs/serving/realtime_api.md` | Realtime audio WS | +| `../../vllm-omni/docs/serving/realtime_duplex_api.md` | Duplex realtime | +| `../../vllm-omni/docs/serving/full_duplex_api.md` | Full-duplex API | +| `../../vllm-omni/docs/models/supported_models.md` | Model × hardware table | +| `../../vllm-omni/docs/contributing/ci/test_execution_guide.md` | How to run tests | +| `../../vllm-omni/docs/contributing/model/adding_tts_model.md` | TTS model guide | +| `../../vllm-omni/docs/contributing/model/adding_omni_model.md` | Omni model guide | +| `../../vllm-omni/docs/contributing/model/adding_diffusion_model.md` | Diffusion model guide | +| `../../vllm-omni/tests/conftest.py` | Test bootstrap | +| `../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_base.py` | TTS e2e | +| `../../vllm-omni/tests/e2e/online_serving/test_pi0_expansion.py` | π0 e2e | +| `../../vllm-omni/tests/e2e/online_serving/test_gr00t_openpi_expansion.py` | GR00T e2e | +| `../../vllm-omni/.buildkite/cuda/test-ready.yml` | L1/L2 CI | +| `../../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/end2end.py` | Offline TTS example | +| `../../vllm-omni/examples/online_serving/pi0/openpi_client.py` | π0 client | +| `../../vllm-omni/examples/online_serving/dreamzero/openpi_client.py` | DreamZero client | +| `../../vllm-omni/examples/offline_inference/qwen3_omni/end2end_async_chunk.py` | Async chunk example | +| `../../vllm-omni/examples/online_serving/qwen3_omni/openai_realtime_client.py` | Realtime client | +| `../../vllm-omni/examples/offline_inference/internvla_a1/end2end.py` | InternVLA example | +| `../../vllm-omni/examples/offline_inference/custom_pipeline/image_to_image/custom_pipeline.py` | Custom pipeline example | +| `../../vllm-omni/benchmarks/tts/bench_tts.py` | TTS benchmark | +| `../../vllm-omni/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md` | DGX Spark recipe (aarch64 hint) | +| `../../vllm-omni/apps/ComfyUI-vLLM-Omni` | ComfyUI app | diff --git a/docs/engines/vllm_omni_models.md b/docs/engines/vllm_omni_models.md new file mode 100644 index 000000000..f8224d5d7 --- /dev/null +++ b/docs/engines/vllm_omni_models.md @@ -0,0 +1,512 @@ +# vLLM-Omni — TTS and VLA Model-Level Analysis + +| Field | Value | +|---|---| +| Local path | `/data/zhoutaichang/embedding_infer/vllm-omni` | +| Remote | `https://github.com/vllm-project/vllm-omni.git` (origin) | +| Branch | `main` | +| HEAD | `7be014bce6374f06c95b703763bdbac4c6198f31` (expected `7be014bc` — matches) | +| Dirty status | clean (`git status --short` empty) | +| Submodules | none (`git submodule status` empty, no `.gitmodules`) | +| Repo AGENTS.md/CLAUDE.md | none at repo root; `.claude/skills/` contains contributor skills only | +| Analysis date | 2026-09-07 (UTC) | +| Scope | Model-level deep dive of TTS and VLA/world-model implementations, feeding `tts_edge_design.md` / `vla_edge_design.md` | + +Evidence labels: **[Verified]** = read in the cited source; **[Inferred]** = reasoned from structure; **[Proposal]** = design suggestion; **[Unknown]** = could not determine. Line numbers (`#Lnnn`) are from the cited checkout. Delegated read-outs by subagents were spot-checked against the tree; link existence was verified mechanically (see §8). + +--- + +## 0. Summary + +- vLLM-Omni is a **serving framework**, not an execution runtime: it composes vLLM's AR runtime (paged KV, continuous batching, CUDA graphs) with two additional stage types — `LLM_GENERATION` (non-AR "generation worker", used for codec decoders/vocoders) and `DIFFUSION` (its own diffusion worker) — orchestrated as multi-stage pipelines ([`stage_config.py#L194`](../../vllm-omni/vllm_omni/config/stage_config.py#L194), [`architecture_overview.md`](../../vllm-omni/docs/design/architecture_overview.md)). [Verified] +- Model topology is code (`PipelineConfig` in each `models//pipeline.py`, registered in [`pipeline_registry.py`](../../vllm-omni/vllm_omni/config/pipeline_registry.py)); placement/resources are YAML in [`vllm_omni/deploy/`](../../vllm-omni/vllm_omni/deploy) (90 files). There are **no** stage YAMLs under `vllm_omni/config/`. [Verified] +- 40 directories under `model_executor/models` (39 model families, every one shipping a `pipeline.py`, plus `common/` for shared kernels/modules; `output_templates.py` defines `OmniOutput`); by primary use: 15 TTS (+ Ming-flash-omni's TTS variant), 12 speech-to-speech/omni, 1 music, 5 image, 4 video/world, 2 VLA (+ `pi0` and `internvla_a1` living under `diffusion/models`). See §1. [Verified] +- **Dominant TTS pattern**: stage 0 = AR talker on a vLLM `Qwen2/Qwen3Model` backbone producing multi-codebook codec tokens (often with a per-frame sub-AR "code predictor"/"depth transformer"), stage 1 = codec decoder on the generation worker, connected by a `SharedMemoryConnector` carrying msgpack-serialised **CPU** tensors. [Verified] +- Streaming ("async chunk") is real and per-family: Qwen3-TTS emits the first chunk after **1 frame (80 ms)**, Fish after 4 frames, CosyVoice3 after 15+3 tokens with **cumulative re-decode**, MOSS-Delay variants effectively **not streaming** (`chunk_frames = 1<<30`), MOSS-Local after 1 frame. [Verified] +- Codec state across chunks is explicit and model-owned: Qwen3-TTS keeps 2-frame/12-frame conv contexts + a 72-frame sliding-window KV; MOSS keeps ring-KV slots; CosyVoice3 keeps cumulative mel; Fish keeps nothing (25-frame overlap re-decode). [Verified] +- **VLA models bypass vLLM entirely**: GR00T N1.7 and π0 are single `DIFFUSION`-stage wrappers around plain HF/torch models; observations arrive as raw numpy dicts in `OmniDiffusionSamplingParams.extra_args["robot_obs"]` over an OpenPI-compatible msgpack WebSocket (`/v1/realtime/robot/openpi`), actions return as `multimodal_output["actions"]`. No paged attention, no compile, `enforce_eager: true`, batch = 1. [Verified] +- DreamZero and LingBot-World are **AR-diffusion world models** with an experimental engine (`ARDiffusionEngine`) owning per-session paged KV; DreamZero = Wan-style 14B causal DiT + UMT5-XXL + CLIP ViT-H + Wan VAE, emitting a 24×8 action chunk per tick. Far outside edge budgets. [Verified] +- Edge-relevant small models in tree (label sources in §1): MOSS-TTS-Nano 0.1B, Gepard ~0.5B, CosyVoice3 0.5B, Ming-TTS 0.5B, Qwen3-TTS 0.6B/1.7B, MOSS-TTS-Realtime 1.7B, VoxCPM2 2B (L4-verified), Audex 2B, GR00T-N1.7 3B, π0 (~2B+0.3B+SigLIP). [Verified labels, sizes not computed] +- Portable sub-modules (plain torch, exportable in principle): Qwen3-TTS 12 Hz tokenizer decoder, Fish DAC decoder, CosyVoice3 CFM + HiFT, MOSS codec (transformer-only), GR00T policy + DiT head, π0 model. Non-portable: every AR talker (vLLM `Qwen3Model`/paged attention), CUDA-graph wrappers, TensorRT paths, Triton Snake kernels (with torch fallback). See §5. [Verified/Inferred] +- Platforms actually in tree: CUDA, ROCm, Ascend NPU, Intel XPU, MUSA ([`platforms/`](../../vllm-omni/vllm_omni/platforms)); **no CPU platform**, no Apple/Android/Jetson-specific code; `requirements/cpu.txt` exists only for docs builds. [Verified] +- Dependency floor: `vllm==0.28.0`, `transformers>=5.10.1,<5.15`, `diffusers==0.40.0`, `torchaudio` only pinned for NPU, `soundfile`, `onnxruntime`, `openai-whisper`; optional `flash-attn-4`, `tensorrt` (CosyVoice3/Step-Audio2), NeMo (Gepard/IndexTTS2), `fish-speech` pip package. [Verified] +- Repo-reported (not reproduced) TTS numbers: Qwen3-TTS TTFP 64 ms / RTF 0.16 on H200 at c=1 ([`qwen3_omni_tts_performance_optimization.md`](../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md)); async-chunk cuts Qwen3-Omni TTFP 6.5 s→0.52 s on H800 ([`async_chunk.md`](../../vllm-omni/docs/design/feature/async_chunk.md)). +- Key gaps for an edge port: no CPU/mobile backend, model code entangled with vLLM layer classes for all talkers, per-family bespoke stage processors, control loops with no KV reuse across steps (GR00T/π0 recompute full VLM prefill every tick), and several documented-but-unimplemented knobs (§7). + +--- + +## 1. Model families inventory + +Every directory under [`vllm_omni/model_executor/models`](../../vllm-omni/vllm_omni/model_executor/models). Stage notation: `id:model_stage(exec, arch, engine_out→final_out)`. Pipeline symbols are in each family's `pipeline.py`; registry keys from [`pipeline_registry.py#L130`](../../vllm-omni/vllm_omni/config/pipeline_registry.py#L130). Async flag from the family's default deploy YAML. Parameter counts are **labels** (HF id / comment / recipe), never computed. [Verified unless marked] + +### 1a. TTS (text → speech) + +| Family | in → out | Architecture type | Stages (async) | Example | Key files | Codec / vocoder | Params (source) | +|---|---|---|---|---|---|---|---| +| [`qwen3_tts`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts) | text+instruct+ref-audio/speaker → 24 kHz | AR (Qwen3) + code predictor + codec | `0:qwen3_tts(LLM_AR, Qwen3TTSTalkerForConditionalGeneration, latent)`, `1:code2wav(LLM_GENERATION, Qwen3TTSCode2Wav, audio→audio)` (async **true**) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts), [online](../../vllm-omni/examples/online_serving/text_to_speech/qwen3_tts) | [`qwen3_tts_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py), [`qwen3_tts_code2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_code2wav.py), [`prompt_embeds_builder.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/prompt_embeds_builder.py) | 12 Hz Mimi-style RVQ (16×2048) + causal conv/Snake decoder | 0.6B / 1.7B (HF ids, [recipe](../../vllm-omni/recipes/Qwen/Qwen3-TTS.md)) | +| [`cosyvoice3`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3) | text+ref-audio+ref-text → 24 kHz | AR (Qwen2-0.5B) → CFM DiT → HiFT | `0:cosyvoice3_talker(LLM_AR, CosyVoice3Model, latent)`, `1:cosyvoice3_code2wav(LLM_GENERATION, latent→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/cosyvoice3), [online](../../vllm-omni/examples/online_serving/text_to_speech/cosyvoice3) | [`cosyvoice3.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py), [`cfm.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py), [`hifigan.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py) | 25 Hz FSQ tokens → 50 Hz mel (CFM, 10 Euler steps) → HiFT ISTFT | 0.5B (HF id) | +| [`fish_speech`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech) | text+ref-audio+ref-text → 44.1 kHz | dual-AR (Qwen3 slow + 4-layer fast) + DAC | `0:fish_speech_slow_ar(LLM_AR, FishSpeechSlowARForConditionalGeneration, latent)`, `1:dac_decoder(LLM_GENERATION, FishSpeechDACDecoder, audio→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/fish_speech), [online](../../vllm-omni/examples/online_serving/text_to_speech/fish_speech) | [`fish_speech_slow_ar.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_slow_ar.py), [`fish_speech_fast_ar.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_fast_ar.py), [`fish_speech_dac_decoder.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_dac_decoder.py) | DAC (1 semantic 4096 + 9 residual 1024), hop 2048 ≈ 21.5 Hz | "4B" ([`speech_api.md`](../../vllm-omni/docs/serving/speech_api.md)) | +| [`moss_tts`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts) | text+ref-audio/instruction → 24 kHz (Local: 48 kHz stereo) | AR (Qwen3 + n_vq+1 heads, delay pattern; Realtime/Local: depth transformer) + transformer codec | `0:moss_tts(LLM_AR, MossTTSDelayModel\|MossTTSRealtime\|MossTTSLocalModel, latent)`, `1:moss_tts_codec(LLM_GENERATION, MossTTSCodecDecoder, audio→audio, retains_state_across_chunks)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/moss_tts) | [`modeling_moss_tts_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py), [`modeling_moss_tts_codec.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py), [`audio_tokenizer_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py) | MOSS Audio Tokenizer (RLFQ, 12.5 Hz, transformer-only) | 8B / 1.7B ([`registry.py#L364`](../../vllm-omni/vllm_omni/model_executor/models/registry.py#L364), YAML headers) | +| [`moss_tts_nano`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano) | text(+ref) → speech | single-stage AR (HF remote code) | `0:moss_tts_nano(LLM_AR, MossTTSNanoForCausalLM, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/moss_tts_nano), [online](../../vllm-omni/examples/online_serving/text_to_speech/moss_tts_nano) | [`modeling_moss_tts_nano.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano/modeling_moss_tts_nano.py) | MOSS-Audio-Tokenizer-Nano | 0.1B ([`pipeline.py#L5`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano/pipeline.py#L5)) | +| [`dots_tts`](../../vllm-omni/vllm_omni/model_executor/models/dots_tts) | text+x-vector → 48 kHz | single-stage AR (Qwen2.5-1.5B) + DiTAR/MeanFlow + AudioVAE + BigVGAN | `0:latent_generator(LLM_AR, DotsTTSForConditionalGeneration, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/dots_tts) | [`dots_tts_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/dots_tts/dots_tts_talker.py), [`dots_tts_vocoder.py`](../../vllm-omni/vllm_omni/model_executor/models/dots_tts/dots_tts_vocoder.py) | AudioVAE → BigVGAN 48 kHz | ~1.7B ([recipe](../../vllm-omni/recipes/rednote-hilab/dots.tts.md)) | +| [`gepard`](../../vllm-omni/vllm_omni/model_executor/models/gepard) | text → 22 kHz | single-stage AR (Qwen3.5 + 32 FSQ heads) | `0:gepard(LLM_AR, GepardTalkerForConditionalGeneration, audio→audio)` (async false, eager) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/gepard) | [`gepard_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/gepard/gepard_talker.py), [`nanocodec.py`](../../vllm-omni/vllm_omni/model_executor/models/gepard/nanocodec.py) | NeMo NanoCodec (external dep) | ~0.5B ([`gepard.yaml#L1`](../../vllm-omni/vllm_omni/deploy/gepard.yaml#L1)) | +| [`glm_tts`](../../vllm-omni/vllm_omni/model_executor/models/glm_tts) | text+ref-audio (WhisperVQ) → 24/32 kHz | AR → CFM DiT → HiFT/Vocos | `0:glm_tts(LLM_AR, GLMTTSForConditionalGeneration, latent)`, `1:glm_tts_dit(LLM_GENERATION, latent→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/glm_tts), [online](../../vllm-omni/examples/online_serving/text_to_speech/glm_tts) | [`glm_tts.py`](../../vllm-omni/vllm_omni/model_executor/models/glm_tts/glm_tts.py), [`glm_tts_dit.py`](../../vllm-omni/vllm_omni/model_executor/models/glm_tts/glm_tts_dit.py), [`whisper_vq.py`](../../vllm-omni/vllm_omni/model_executor/models/common/whisper_vq.py) | HiFT (reused from CosyVoice3) or Vocos2D TorchScript | Unknown | +| [`higgs_audio_v2`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v2) | text → 24 kHz | AR (Llama-3.2-3B, 8 codebooks, DualFFN) + RVQ/DAC | `0:higgs_audio_v2(LLM_AR, latent)`, `1:code2wav(LLM_GENERATION, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/higgs_audio_v2) | [`higgs_audio_v2_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v2/higgs_audio_v2_talker.py), [`higgs_audio_decoder.py`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v2/higgs_audio_decoder.py) | RVQ 8×1024 + DAC decoder, 25 fps | 3B (HF id) | +| [`higgs_audio_v3`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v3) | text+ref → 24 kHz | AR (Qwen3 ~4B, delay pattern) + RVQ/DAC | same shape, key `higgs_multimodal_qwen3` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/higgs_audio_v3), [online](../../vllm-omni/examples/online_serving/text_to_speech/higgs_audio_v3) | [`higgs_audio_v3_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py) | v2 decoder | ~4B (talker docstring) | +| [`indextts2`](../../vllm-omni/vllm_omni/model_executor/models/indextts2) | text+ref(+emotion ref) → speech | GPT2-style AR → S2Mel CFM (25 steps) + BigVGAN | `0:indextts2_talker(LLM_AR, latent)`, `1:indextts2_s2mel_decoder(LLM_GENERATION, audio→audio)` (async **false**, non-streaming) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/indextts2), [online](../../vllm-omni/examples/online_serving/text_to_speech/indextts2) | [`indextts2_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/indextts2/indextts2_talker.py), [`indextts2_s2mel_decoder.py`](../../vllm-omni/vllm_omni/model_executor/models/indextts2/indextts2_s2mel_decoder.py) | S2Mel CFM → BigVGAN; w2v-bert (~600M) semantic front-end | Unknown | +| [`ming_tts`](../../vllm-omni/vllm_omni/model_executor/models/ming_tts) | text+ref (CAM++) → 44.1 kHz | AR LM + per-step CFM head (DiTAR) → AudioVAE | `0:llm(LLM_AR, MingTTSForConditionalGeneration, latent)`, `1:audio_vae(LLM_GENERATION, audio→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/ming_tts), [online](../../vllm-omni/examples/online_serving/text_to_speech/ming_tts) | [`ming_tts.py`](../../vllm-omni/vllm_omni/model_executor/models/ming_tts/ming_tts.py), [`common/ming/cfm_dit.py`](../../vllm-omni/vllm_omni/model_executor/models/common/ming/cfm_dit.py) | AudioVAE (Qwen2 decoder + ISTFT) | 0.5B dense / 16.8B-A3B MoE ([`pipeline.py#L18`](../../vllm-omni/vllm_omni/model_executor/models/ming_tts/pipeline.py#L18)) | +| [`omnivoice`](../../vllm-omni/vllm_omni/model_executor/models/omnivoice) | text+ref → 24 kHz | non-AR masked-generative (Qwen3 backbone) → DAC | `0:dit(DIFFUSION, OmniVoicePipeline, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/omnivoice), [online](../../vllm-omni/examples/online_serving/text_to_speech/omnivoice) | [`omnivoice_generator.py`](../../vllm-omni/vllm_omni/model_executor/models/omnivoice/omnivoice_generator.py), [`fused_qkv_rope.py`](../../vllm-omni/vllm_omni/model_executor/models/omnivoice/fused_qkv_rope.py) (Triton) | RVQ + `transformers` DacModel | Unknown | +| [`voxcpm2`](../../vllm-omni/vllm_omni/model_executor/models/voxcpm2) | text+ref → 48 kHz | single-stage AR (MiniCPM4 paged) + LocDiT CFM + AudioVAE | `0:latent_generator(LLM_AR, VoxCPM2TalkerForConditionalGeneration, audio→audio)`, custom scheduler (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/voxcpm2), [online](../../vllm-omni/examples/online_serving/text_to_speech/voxcpm2) | [`voxcpm2_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/voxcpm2/voxcpm2_talker.py), [`scheduler.py`](../../vllm-omni/vllm_omni/model_executor/models/voxcpm2/scheduler.py) | LocDiT CFM → AudioVAE | 2B ([recipe](../../vllm-omni/recipes/OpenBMB/VoxCPM2.md)); L4-24GB target ([`voxcpm2.yaml#L2`](../../vllm-omni/vllm_omni/deploy/voxcpm2.yaml#L2)) | +| [`voxtral_tts`](../../vllm-omni/vllm_omni/model_executor/models/voxtral_tts) | text+voice → 24 kHz | AR LM → flow-matching acoustic transformer + codec | `0:audio_generation(LLM_AR, latent)`, `1:audio_tokenizer(LLM_GENERATION, audio→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/voxtral_tts), [online](../../vllm-omni/examples/online_serving/text_to_speech/voxtral_tts) | [`voxtral_tts_audio_generation.py`](../../vllm-omni/vllm_omni/model_executor/models/voxtral_tts/voxtral_tts_audio_generation.py) | Mistral audio tokenizer (Euler ODE + CFG) | 4B (HF id) | +| [`ming_flash_omni`](../../vllm-omni/vllm_omni/model_executor/models/ming_flash_omni) (TTS variant `ming_flash_omni_tts`) | text → speech | single `LLM_GENERATION` talker (Qwen2 + CFM + AudioVAE) | `0:ming_tts(LLM_GENERATION, MingFlashOmniTalkerForConditionalGeneration, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/ming_flash_omni_tts) | [`ming_flash_omni_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/ming_flash_omni/ming_flash_omni_talker.py) | CFM DiT + AudioVAE | Unknown | + +### 1b. Speech-to-speech / omni (multimodal in → text + speech) + +| Family | in → out | Architecture | Stages (async) | Example | Key files | Vocoder | Params | +|---|---|---|---|---|---|---|---| +| [`qwen2_5_omni`](../../vllm-omni/vllm_omni/model_executor/models/qwen2_5_omni) | text/image/audio/video → text+speech | thinker AR → talker AR → DiT+BigVGAN | `0:thinker(LLM_AR)`, `1:talker(LLM_AR)`, `2:code2wav(LLM_GENERATION)` (async false) | [offline](../../vllm-omni/examples/offline_inference/qwen2_5_omni) | [`qwen2_5_omni_token2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_token2wav.py) | DiT (50 Hz) + BigVGAN | 7B / 3B | +| [`qwen3_omni`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_omni) | same → text+16 kHz speech | MoE thinker → MoE talker (+code predictor) → ConvNeXt Code2Wav | 3 stages, async **true** (reference streaming impl) | [offline](../../vllm-omni/examples/offline_inference/qwen3_omni), [online](../../vllm-omni/examples/online_serving/qwen3_omni) | [`qwen3_omni_code2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_code2wav.py) | ~200M ConvNet ([doc](../../vllm-omni/docs/design/architecture_overview.md)) | 30B-A3B | +| [`minicpmo_4_5`](../../vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5) | omni → text+24 kHz; **full-duplex** | thinker AR → TTS AR → CFM+HiFT | 3 stages, async true, `duplex_control_enabled` | [offline](../../vllm-omni/examples/offline_inference/minicpmo), [online](../../vllm-omni/examples/online_serving/minicpmo) | [`minicpmo_4_5_omni_llm.py`](../../vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_llm.py), [`duplex/stage0.py`](../../vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py) | CosyVoice-style CFM + HiFT | Unknown | +| [`step_audio2`](../../vllm-omni/vllm_omni/model_executor/models/step_audio2) | audio+text → text+speech | AR thinker → CosyVoice2 flow + HiFT | 2 stages (async false default; `_async_chunk.yaml` true) | [offline](../../vllm-omni/examples/offline_inference/step_audio2) | [`step_audio2_token2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/step_audio2/step_audio2_token2wav.py), [`step_audio2_dit_trt.py`](../../vllm-omni/vllm_omni/model_executor/models/step_audio2/step_audio2_dit_trt.py) | CosyVoice2 DiT (+TRT) + HiFT | Unknown | +| [`mimo_audio`](../../vllm-omni/vllm_omni/model_executor/models/mimo_audio) | audio+text → text+24 kHz | fused thinker/talker AR + patch decoder → Vocos | 2 stages, async true; RTX 5090 profile | [offline](../../vllm-omni/examples/offline_inference/mimo_audio) | [`mimo_audio_code2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/mimo_audio/mimo_audio_code2wav.py) | VQ → Vocos ISTFT | 7B | +| [`covo_audio`](../../vllm-omni/vllm_omni/model_executor/models/covo_audio) | audio+text → text+24 kHz | fused AR → CFM + BigVGAN | 2 stages, async false | [offline](../../vllm-omni/examples/offline_inference/covo_audio) | [`token2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/covo_audio/token2wav.py) | CFM + BigVGAN | 7B | +| [`dynin_omni`](../../vllm-omni/vllm_omni/model_executor/models/dynin_omni) | text → text+image+audio | 3× `LLM_GENERATION` chain | async false; L4-sized CI profile | [offline](../../vllm-omni/examples/offline_inference/dynin_omni) | [`dynin_omni_common.py`](../../vllm-omni/vllm_omni/model_executor/models/dynin_omni/dynin_omni_common.py) | external tokenizer | Unknown | +| [`ming_flash_omni`](../../vllm-omni/vllm_omni/model_executor/models/ming_flash_omni) | omni → text/speech/image | BailingMoE-v2 AR (TP4) → talker / DiT | async false | [offline](../../vllm-omni/examples/offline_inference/ming_flash_omni) | [`modeling_bailing_moe_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/ming_flash_omni/modeling_bailing_moe_v2.py) | CFM+AudioVAE | Unknown (MoE) | +| [`aura_omni`](../../vllm-omni/vllm_omni/model_executor/models/aura_omni) | mic audio+video → text+speech | 4-stage cascade ASR→VLM→Qwen3-TTS→Code2Wav | async false, single GPU | [offline](../../vllm-omni/examples/offline_inference/aura_omni) | [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/aura_omni/pipeline.py) | Qwen3-TTS Code2Wav | 1.7B+?+1.7B | +| [`audex`](../../vllm-omni/vllm_omni/model_executor/models/audex) | text→16 kHz (TTS), audio→text/speech (S2S), caption→audio (TTA) | Nemotron AR + streaming causal speech decoder / XCodec1 | 4 pipelines; TTS/S2S async true | [offline](../../vllm-omni/examples/offline_inference/audex) | [`speech_decoder/modeling_audex_causal_speech_decoder.py`](../../vllm-omni/vllm_omni/model_executor/models/audex/speech_decoder/modeling_audex_causal_speech_decoder.py) | Audex causal decoder (4-frame lookahead) | 2B / 30B-A3B ([recipe](../../vllm-omni/recipes/NVIDIA/Nemotron-Labs-Audex.md)) | +| [`nemotron_voicechat`](../../vllm-omni/vllm_omni/model_executor/models/nemotron_voicechat) | speech → text+22.05 kHz; full-duplex | FastConformer + NemotronH AR → EAR-TTS AR → RVQ-VAE | 3 stages; `_duplex.yaml` | [offline](../../vllm-omni/examples/offline_inference/nemotron_voicechat) | [`nemo_vendored/ear_tts_model.py`](../../vllm-omni/vllm_omni/model_executor/models/nemotron_voicechat/nemo_vendored/ear_tts_model.py) | NeMo RVQ-VAE (fp32) | 11B | +| [`personaplex`](../../vllm-omni/vllm_omni/model_executor/models/personaplex) | audio → audio (Moshi finetune, full-duplex) | Helium temporal AR + depformer + Mimi | 2 stages, async true | [online](../../vllm-omni/examples/online_serving/personaplex) | [`personaplex_depformer.py`](../../vllm-omni/vllm_omni/model_executor/models/personaplex/personaplex_depformer.py), [`personaplex_mimi.py`](../../vllm-omni/vllm_omni/model_executor/models/personaplex/personaplex_mimi.py) | Mimi 24 kHz | Unknown | + +### 1c. Music, image, video / world + +| Family | in → out | Architecture | Stages | Key files | Params | +|---|---|---|---|---|---| +| [`minimax_music3`](../../vllm-omni/vllm_omni/model_executor/models/minimax_music3) | caption+lyrics → 32 kHz stereo music | AR (Qwen3 + depth decoder, 8 RVQ, mandatory CFG) → CFM DiT + DAV | 2 stages, async true, 140 GB card | [`talker.py`](../../vllm-omni/vllm_omni/model_executor/models/minimax_music3/talker.py), [`dit.py`](../../vllm-omni/vllm_omni/model_executor/models/minimax_music3/dit.py) | Unknown | +| [`bagel`](../../vllm-omni/vllm_omni/model_executor/models/bagel) | text+image → text/image | AR thinker (KV transfer) → DiT; single-stage variant | 3 pipelines | [`bagel.py`](../../vllm-omni/vllm_omni/model_executor/models/bagel/bagel.py) | 7B | +| [`glm_image`](../../vllm-omni/vllm_omni/model_executor/models/glm_image) | text → image | AR VQ tokens → DiT | 2 stages, 2 GPUs | [`glm_image_ar.py`](../../vllm-omni/vllm_omni/model_executor/models/glm_image/glm_image_ar.py) | Unknown | +| [`hunyuan_image3`](../../vllm-omni/vllm_omni/model_executor/models/hunyuan_image3) | text+image → text+image | MoE AR → DiT (KV reuse) | 3 pipelines, TP2–4 | [`hunyuan_image3.py`](../../vllm-omni/vllm_omni/model_executor/models/hunyuan_image3/hunyuan_image3.py) | 80B (comment) | +| [`mammoth_moda2`](../../vllm-omni/vllm_omni/model_executor/models/mammoth_moda2) | text+image → image | AR (Qwen2.5-VL) → DiT on generation worker | 2 stages | [`mammoth_moda2.py`](../../vllm-omni/vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py) | Unknown | +| [`lance`](../../vllm-omni/vllm_omni/model_executor/models/lance) | text+image → image/text/video | single DiT stage (embedded Qwen2-MoT + Wan VAE) | 1 stage | [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/lance/pipeline.py) | 3B (recipe) | +| [`wan2_2`](../../vllm-omni/vllm_omni/model_executor/models/wan2_2) | text(+image) → video | diffusion DiT | 1 stage | [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/wan2_2/pipeline.py) | 5B (YAML) | +| [`hunyuan_video`](../../vllm-omni/vllm_omni/model_executor/models/hunyuan_video) | text/image → video | diffusion DiT | 1 stage | [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/hunyuan_video/pipeline.py) | Unknown | +| [`minimax_h3`](../../vllm-omni/vllm_omni/model_executor/models/minimax_h3) | text+refs → video+audio | Qwen3-VL text encoder (LLM_AR) → inline DiT | 2 stages, 6 GPUs default | [`text_encoder.py`](../../vllm-omni/vllm_omni/model_executor/models/minimax_h3/text_encoder.py) | Unknown | +| [`lingbot_world`](../../vllm-omni/vllm_omni/model_executor/models/lingbot_world) | image+text+camera controls → streaming video | causal AR-diffusion DiT, paged KV (world model) | 1 DIFFUSION stage, `ARDiffusionEngine` | [`diffusion/models/lingbot_world/pipeline.py`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/pipeline.py) | 14B (HF id) | + +### 1d. VLA / robot policy + +| Family | in → out | Architecture | Stages | Example | Key files | Params | +|---|---|---|---|---|---|---| +| [`gr00t`](../../vllm-omni/vllm_omni/model_executor/models/gr00t) | images+state+instruction → action chunk | Qwen3-VL (Cosmos-Reason2-2B, 12 layers) + flow-matching DiT head | `0:diffusion(DIFFUSION, Gr00tN1d7Pipeline, →actions)`; `max_num_seqs 1`, eager | none in `examples/` (recipe only) | [`pipeline_gr00t.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py), [`policy.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py), [`gr00t_n1d7.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py) | 3B (HF id) | +| [`dreamzero`](../../vllm-omni/vllm_omni/model_executor/models/dreamzero) | images+state+text → actions (+video latents) | Wan-style causal 14B DiT world model + UMT5-XXL + CLIP ViT-H + Wan VAE | `0:diffusion(DIFFUSION, DreamZeroPipeline, →image)`; `ARDiffusionEngine`; CFG-parallel 2 | [offline](../../vllm-omni/examples/offline_inference/dreamzero), [online](../../vllm-omni/examples/online_serving/dreamzero) | [`pipeline_dreamzero.py`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/pipeline_dreamzero.py), [`causal_wan_model.py`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/causal_wan_model.py) | 14B (comment [`causal_wan_model.py#L743`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/causal_wan_model.py#L743)) | +| `pi0` (under `diffusion/models`) | 3 images+state+prompt → 50×32 actions | PaliGemma (SigLIP+Gemma-2B) + Gemma-300M action expert, flow matching | `0:diffusion(DIFFUSION, Pi0Pipeline, →action)` ([`pi0_pipeline_config.py`](../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py)) | [online](../../vllm-omni/examples/online_serving/pi0) | [`pipeline_pi0.py`](../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py), [`modeling_pi0.py`](../../vllm-omni/vllm_omni/diffusion/models/pi0/modeling_pi0.py) | "~13 GB checkpoint" ([recipe](../../vllm-omni/recipes/lerobot/Pi0.md)) | +| `internvla_a1` (diffusion only, no PipelineConfig) | 3 cams×2 frames+state+task → 50×32 actions | Qwen3-VL und/gen/act three-expert + Cosmos CI8x8 tokenizer, flow matching | offline-only via `pipeline.forward` | [offline](../../vllm-omni/examples/offline_inference/internvla_a1) | [`pipeline_internvla_a1.py`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/pipeline_internvla_a1.py), [`model_internvla_a1.py`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py) | 3B (HF id) | + +### 1e. Shared `common/` modules [Verified] + +[`common/`](../../vllm-omni/vllm_omni/model_executor/models/common): [`snake_activation.py`](../../vllm-omni/vllm_omni/model_executor/models/common/snake_activation.py) (Snake/SnakeBeta, lazy `@triton.jit` on CUDA with torch fallback; used by Qwen3-TTS, Qwen3-Omni, Qwen2.5-Omni, CoVo, CosyVoice3); [`alias_free_activation.py`](../../vllm-omni/vllm_omni/model_executor/models/common/alias_free_activation.py) (plain torch); [`qwen3_code_predictor.py`](../../vllm-omni/vllm_omni/model_executor/models/common/qwen3_code_predictor.py) (shared code predictor, SDPA + `torch.compile` + optional manual CUDA graphs); [`nucleus_ras_sampling.py`](../../vllm-omni/vllm_omni/model_executor/models/common/nucleus_ras_sampling.py) (used by GLM-TTS only; CosyVoice3 has its own copy); [`whisper_vq.py`](../../vllm-omni/vllm_omni/model_executor/models/common/whisper_vq.py); [`cfg_pairing.py`](../../vllm-omni/vllm_omni/model_executor/models/common/cfg_pairing.py) (CFG request pairing for the vLLM scheduler); [`ops/fused_group_norm_silu.py`](../../vllm-omni/vllm_omni/model_executor/models/common/ops/fused_group_norm_silu.py) and [`ops/fused_adaptive_group_norm_silu.py`](../../vllm-omni/vllm_omni/model_executor/models/common/ops/fused_adaptive_group_norm_silu.py) (Triton `@triton.autotune`/`@triton.jit`, torch fallback when `HAS_TRITON` false); [`ming/`](../../vllm-omni/vllm_omni/model_executor/models/common/ming) (CFM DiT, AudioVAE, batch-1 CUDA-graph executor `cfm_cudagraph.py`, CAM++ via onnxruntime). + +--- + +## 2. TTS deep dive + +Four families chosen for documentation depth and architectural spread: **Qwen3-TTS** (AR + code predictor + neural codec, streaming reference), **CosyVoice3** (AR + flow-matching + HiFT), **Fish Speech S2 Pro** (dual-AR + DAC), **MOSS-TTS** (delay-pattern / depth-transformer + transformer codec). Cross-family mechanics are in §4. + +### 2.1 Qwen3-TTS (`qwen3_tts`) + +**Pipeline** [Verified] — [`pipeline.py#L16`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/pipeline.py#L16): stage 0 `qwen3_tts` (`LLM_AR`, `Qwen3TTSTalkerForConditionalGeneration`, `engine_output_type="latent"`, `owns_tokenizer`, `sampling_constraints={"detokenize": False, "stop_token_ids": [2150]}`, async func `talker2code2wav_async_chunk`, full-payload func `talker2code2wav_full_payload`); stage 1 `code2wav` (`LLM_GENERATION`, `Qwen3TTSCode2Wav`, `final_output_type="audio"`, `sync_process_input_func=talker2code2wav_token_only`, `requires_full_payload_input`, `extras.tts_args.max_instructions_length=500`). Deploy [`qwen3_tts.yaml`](../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml): `async_chunk: true`; connector extras `codec_chunk_frames: 25`, `codec_left_context_frames: 72` ("must match the decoder sliding attention window"), `initial_codec_chunk_frames: 1`, `decode_batch_max_size: 4`; stage 0 `max_num_seqs 64`, `gpu_memory_utilization 0.3`, `max_model_len 4096`, sampling `temperature 0.9, top_k 50, repetition_penalty 1.05, min_tokens 2`, `subtalker_sampling_params {do_sample, 0.9, top_k 50}`; stage 1 `max_num_seqs 64`, `enforce_eager: false`, `max_model_len 65536` ("prefill length is Q × num_frames"); `platforms.rocm` forces stage 1 eager (MIOpen Conv1d inside graph capture), `platforms.npu` uses `cudagraph_mode: PIECEWISE`. High-concurrency 2-GPU profile: [`qwen3_tts_high_concurrency.yaml`](../../vllm-omni/vllm_omni/deploy/qwen3_tts_high_concurrency.yaml). + +**Conditioning** [Verified] — [`prompt_embeds_builder.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/prompt_embeds_builder.py) `Qwen3TTSPromptEmbedsBuilder.build_prompt_embeds()` (L918-1455): text via HF `AutoTokenizer` in `<|im_start|>assistant\n{text}<|im_end|>` template, embedded through a separate `text_embedding` table + `text_projection` MLP ([`qwen3_tts_talker.py#L371`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L371)); `instruct` prepended as a user turn; `language` inserted as codec "think" tokens from `talker_config.codec_language_id`; **CustomVoice** = speaker id embedded via codec table; **VoiceDesign** = tags + text only; **Base** (clone) = ECAPA-TDNN speaker encoder `Qwen3TTSSpeakerEncoder` ([`qwen3_tts_talker.py#L202`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L202)) over a 128-mel spectrogram (24 kHz, n_fft 1024, hop 256), optionally plus ICL: reference audio encoded to codec frames by `Qwen3TTSTokenizerV2Encoder` (a `MimiModel` subclass, [`modeling_qwen3_tts_tokenizer_v2.py#L1587`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py#L1587)) in `_encode_ref_audio_batch` ([`#L1039`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L1039)) — 16 codebooks kept, cached by SHA1 of waveform (LRU 256). Audio loading: `soundfile` for base64, vLLM `MediaConnector`/`load_audio` + `AudioResampler` otherwise (no direct librosa/torchaudio import in this dir). Streaming vs non-streaming prompt mode: non-streaming (default for CustomVoice/VoiceDesign) puts full text in prefill; streaming (default for Base) feeds one text-embedding row per decode step from a `trailing_text_hidden` queue ([`#L833`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L833)). + +**Generation** [Verified] — AR over codec **frames**: vLLM samples codebook-0 per step from `lm_head` (logits masked to `[1, codebook_vocab) ∪ {eos}`, [`#L414`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L414)); residual codebooks 1..Q-1 predicted by the **code predictor** ([`common/qwen3_code_predictor.py`](../../vllm-omni/vllm_omni/model_executor/models/common/qwen3_code_predictor.py) `CodePredictorWrapper.forward` L996-1138): a 5-layer / hidden-1024 / 16-head decoder (config defaults [`configuration_qwen3_tts.py#L191`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/configuration_qwen3_tts.py#L191)) that **re-prefills** `[proj(hidden), proj(embed(c0)), …]` per codebook with SDPA and **no KV cache**, Gumbel-max top-k sampling; on CUDA it is `torch.compile(dynamic=False)` with power-of-2 batch buckets, manual CUDA graphs only on NPU ([`qwen3_tts_code_predictor_vllm.py#L38`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_code_predictor_vllm.py#L38)). Runner hook `talker_mtp()` ([`#L1245`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L1245)) returns `(inputs_embeds [B,H], audio_codes [B,Q])`; next-step input embedding = Σ codebook embeddings + one text-step vector. Q = 16 codebooks × 2048 for the 12 Hz family ([`stage_input_processors/qwen3_tts.py#L390`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py#L390)); frame = 1920 samples @ 24 kHz = **80 ms = 12.5 Hz** ([`configuration_qwen3_tts_tokenizer_v2.py#L147`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/configuration_qwen3_tts_tokenizer_v2.py#L147)). The 25 Hz tokenizer ([`tokenizer_25hz/`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_25hz), Whisper-VQ + DiT + BigVGAN, imports onnxruntime/torchaudio-kaldi/flash-attn) is **not** used by the vLLM stages. + +**Codec / vocoder** [Verified] — [`qwen3_tts_code2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_code2wav.py) `Qwen3TTSCode2Wav` wraps `Qwen3TTSTokenizerV2Decoder` ([`modeling_qwen3_tts_tokenizer_v2.py#L807`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py#L807)) in **float32**: `SplitResidualVectorQuantizer.decode` (1 semantic + 15 acoustic) → causal `pre_conv(k=3)` → 8-layer **sliding-window-72 causal transformer** → two ×2 causal ConvTranspose+ConvNeXt upsamples → BigVGAN/DAC-style stack (SnakeBeta, rates 8,5,4,3, dilations 1,3,9) → clamp. Runs on the generation worker (stage 1, `requires_raw_input_tokens`, `requires_request_ids`); input codes arrive flat codebook-major `[Q*F]` in `runtime_additional_information["codes"]["audio"]`; **batched** across requests (`[B,Q,max_F]` + lengths, `decode_batch_max_size`), chunked via `batched_chunked_decode(chunk 300, left_context 25)` in non-async mode, or **stateful per-request** decode in async mode. Output `{"model_outputs": [float32 wav], "sr": [24000]}`. CUDA graphs: [`segmented_graph_wrapper.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/segmented_graph_wrapper.py) captures stateful ICL-prefix / xvec-prefix graphs per batch bucket and "combined suffix" graphs for every reachable chunk length derived from `initial→+25→…→72→72+25` (`_derive_suffix_transitions` L113); the older stateless [`cuda_graph_decoder_wrapper.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py) survives only for its `_batched_chunked_decode` helper. + +**Streaming / first-audio structure** [Verified] — `talker2code2wav_async_chunk` ([`stage_input_processors/qwen3_tts.py#L77`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py#L77)) is called every talker step by `OmniChunkTransferAdapter._send_single_request_for_generation` ([`chunk_transfer_adapter.py#L654`](../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py#L654)); emits when `len == initial_chunk` or `(len-initial) % 25 == 0` or finished; payload = only the **new** frames (`left_context_size=0`, because Code2Wav keeps context per request), first chunk of an ICL request prepends ≤72 reference frames. Dynamic initial chunk from server load (`compute_dynamic_initial_chunk_size`), optional `codec_chunk_ramp [4,4,8,16,25]`. **Before first audio**: text prefill → **1 talker step** (+15 code-predictor sub-steps) → SHM put → Code2Wav first-chunk path (`_decode_xvec_first_chunk` / `_decode_icl_first_chunk`) → 1920 samples. Repo-reported TTFP 64 ms at c=1 on H200 ([doc](../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md), not reproduced). + +**State** [Verified] — Talker KV: vLLM `Qwen3Model` paged attention ([`#L355`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L355)); per-request GPU-resident buffers `hidden_states.last`, `hidden_states.trailing_text`, `codes.audio`, `codes.ref`. Code predictor: stateless. Code2Wav: `_decoder_state_cache[req_id]` holding `ref_hidden`, `suffix_quantized` (≤72 frames), `past_key_values` (`DynamicCache` of the ref prefix; suffix KV recomputed per chunk), and explicit conv contexts `_CONV_CONTEXT_FRAME=2` (pre_conv) and `_DOWNSTREAM_CONTEXT_FRAME=12` (upsample+decoder stack) ([`modeling_qwen3_tts_tokenizer_v2.py#L803`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py#L803)); tests assert these are the minimal exact contexts ([`test_qwen3_tts_incremental_decode.py`](../../vllm-omni/tests/model_executor/models/qwen3_tts/test_qwen3_tts_incremental_decode.py)). + +**Inter-stage tensors** [Verified] — stage 0 `make_omni_output` ([`#L635`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L635)) → `OmniPayload{"codes": {"audio": long [span,Q], "ref": [T_ref,Q]}, "meta": {"ref_code_len", "codec_streaming"}}`, hidden states **not** shipped (`omni_pooler_payload_include_hidden=False`). Wire: `OmniPayloadStruct(codes.audio flat long [Q*F], meta.{finished,left_context_size,ref_context_*})` from [`data_entry_keys.py`](../../vllm-omni/vllm_omni/data_entry_keys.py). Stage 1 → `{"model_outputs": [float32 [n]], "sr": [int32]}`. + +**Forced aligner** [Verified] — [`qwen3_tts_forced_aligner.yaml`](../../vllm-omni/vllm_omni/deploy/qwen3_tts_forced_aligner.yaml) + [`utils/forced_aligner.py#L85`](../../vllm-omni/vllm_omni/utils/forced_aligner.py#L85) `inject_forced_aligner_stage` appends a third `LLM_AR` pooling stage (`Qwen3-ForcedAligner-0.6B`) consuming the waveform + text and emitting per-word `{word,start_ms,end_ms}` for the `word_timestamps` API option. + +### 2.2 CosyVoice3 (`cosyvoice3`) + +**Pipeline** [Verified] — [`pipeline.py#L23`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/pipeline.py#L23): stage 0 `cosyvoice3_talker` (`LLM_AR`, `CosyVoice3Model`, stop `[6562]` = logsumexp-merged 200 stop logits, async `talker2code2wav_async_chunk`, full-payload `text2flow_full_payload`); stage 1 `cosyvoice3_code2wav` (`LLM_GENERATION`, `final_output_type="audio"`, sync `text2flow_token_only`, `requires_full_payload_input`). Deploy [`cosyvoice3.yaml`](../../vllm-omni/vllm_omni/deploy/cosyvoice3.yaml): `async_chunk: true`; connector `codec_chunk_frames: 15`, `codec_left_context_frames: 25` (unused by this family's processor), `codec_vocab_size: 6561`; stage 0 `max_num_seqs 8`, `gpu 0.4`, `enforce_eager: false`, sampling `top_k 25, top_p 0.8, repetition_penalty 1.0001` (near-identity so vLLM tracks output ids for RAS); stage 1 `enforce_eager: true` ("CUDA graphs don't work with dynamic conv shapes"), `gpu 0.2`; both on GPU 0. No `platforms:` section. + +**Conditioning** [Verified] — `CosyVoice3MultiModalProcessor._call_hf_processor` ([`cosyvoice3.py#L248`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L248)): inputs `prompt`, `mm_data["audio"]=(wav,sr)`, `mm_kwargs["prompt_text"]` (system prompt + transcript, e.g. `"You are a helpful assistant.<|endofprompt|>…"`), optional `voice_name` for the speaker cache. No explicit instruction/language field. Prompt speech tokens via `s3tokenizer` (`speech_tokenizer_v3_25hz`, GPU; ONNX path unused, [`#L221`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L221)); speaker embedding = **CAM++** 192-d from kaldi fbank (`torchaudio.compliance.kaldi.fbank`), TensorRT `CampplusTRT` ([`speaker_embedding_trt.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/speaker_embedding_trt.py)) or onnxruntime CPU fallback; prompt mel 80-bin/24 kHz/hop 480 ([`utils.py#L49`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/utils.py#L49)), trimmed to 2:1 mel:token. Audio loading `torchaudio.load(backend="soundfile")` + `torchaudio.transforms.Resample` ([`utils.py#L89`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/utils.py#L89)). Tokenizer: `AutoTokenizer` from `CosyVoice-BlankEN` + special tokens ([`tokenizer.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/tokenizer.py)). + +**Generation** [Verified] — AR over semantic speech tokens, vocab 6561 (=3⁸, FSQ [Inferred]) + 200 stop ids ([`cosyvoice3_talker.py#L125`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3_talker.py#L125)), **25 Hz**, `token_mel_ratio=2`. Backbone `VLLMQwen2Encoder` wrapping vLLM `Qwen2Model` (hidden 896, 24 layers, 14/2 heads → Qwen2.5-0.5B class [Inferred]) with paged attention. Sampling: model-owned `prefer_model_sampler=True` implementing nucleus + **RAS** (repetition-aware: mask `top_id` if it appears ≥ `win_size*tau_r` in last 10 outputs; [`#L610`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L610)); bypassed to vLLM `Sampler` when logprobs/penalties requested. + +**Code2wav** [Verified] — [`cosyvoice3_code2wav.py#L43`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3_code2wav.py#L43): `PreLookaheadLayer` (3-frame right context) → `CausalConditionalCFM` ([`cfm.py#L190`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py#L190)) with estimator `DiT` ([`diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py#L340`](../../vllm-omni/vllm_omni/diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py#L340): dim 1024, depth 22, 16 heads, uses `vllm_omni.diffusion.attention.layer.Attention`, diffusers `AdaLayerNormZero`, `x_transformers` rotary) — **10 Euler steps**, cosine t-schedule, CFG 0.7 as batch-2 → 80-bin mel at 50 Hz → `CausalHiFTGenerator` ([`hifigan.py#L606`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py#L606)): NSF source (f0 predictor pinned to **CPU** for precision, L757) + ISTFTNet (upsample 8·5·3, n_fft 16/hop 4 → 480 samples/frame), Snake activations, forced fp32, 24 kHz. **TensorRT**: `COSYVOICE3_TRT` env (default on) swaps the DiT estimator for a TRT engine built from `flow.decoder.estimator.autocast_fp16.onnx` (downloaded from `yuekai/Fun-CosyVoice3-0.5B-2512-FP16-ONNX` if absent, [`#L901`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L901)) via [`flow_estimator_trt.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/flow_estimator_trt.py); requires the `tensorrt` python package. **Batching**: stage 1 is scheduled with `max_num_seqs 8` but `CosyVoice3Model.forward` loops requests **sequentially** at batch 1 (`inference` asserts `token.shape[0]==1`, [`cfm.py#L302`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py#L302)); only CFG doubles the DiT batch. + +**Streaming** [Verified] — `talker2code2wav_async_chunk` ([`stage_input_processors/cosyvoice3.py#L96`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/cosyvoice3.py#L96)): first chunk after `chunk(15) + prompt_pad + 3 lookahead` tokens, subsequent hops double (15, 30, 60, 60 …; ≈0.6 s, 1.2 s, 2.4 s of audio [Inferred]); payload `codes.audio` = **cumulative prefix** of all tokens so far with `meta.left_context_size` = already-emitted length. Consumer `forward_streaming` ([`cosyvoice3_code2wav.py#L196`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3_code2wav.py#L196)) reruns the **full CFM over prompt + all tokens** and HiFT over cumulative mel, then trims — per-chunk cost grows with history (O(n²) overall) [Inferred]. No waveform cross-fade (`_stitch_stream_audio` is pass-through); `mel_window/speech_window` attributes are dead. First audio requires: mm preprocessing (s3tokenizer GPU + CAM++ + mel) → talker prefill + ≥18+pad decode steps → SHM → CFM (10 steps × batch 2) → HiFT. + +**State** [Verified] — talker: vLLM paged KV only. Code2wav: `_stream_vocoder_cache_by_req[req] = {"mel": cumulative CPU mel, "speech_offset"}` cleared on `stream_finished`; no HiFT conv cache, no CFM noise cache. + +**Inter-stage tensors** [Verified] — prefill emits `embed.{speech_token [1,T] int32, speech_feat [1,2T,80], embedding [1,192], speech_token_len}` once ([`#L992`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L992)); async chunks `codes.audio` int64 `[prefix_len]` + `meta`; stage-1 output `{"audio": [float32 1-D], "sr": [24000]}`. + +### 2.3 Fish Speech S2 Pro (`fish_qwen3_omni`) + +**Pipeline** [Verified] — [`pipeline.py#L21`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/pipeline.py#L21): stage 0 `fish_speech_slow_ar` (`LLM_AR`, `FishSpeechSlowARForConditionalGeneration`, stop `[151645]`, async `slow_ar_to_dac_decoder_async_chunk` — **no sync full-payload func**); stage 1 `dac_decoder` (`LLM_GENERATION`, `FishSpeechDACDecoder`). Deploy [`fish_qwen3_omni.yaml`](../../vllm-omni/vllm_omni/deploy/fish_qwen3_omni.yaml): connector `codec_chunk_frames: 25` (≈1.16 s at ~21 Hz), `codec_left_context_frames: 25`, `initial_codec_chunk_frames: 4`; stage 0 `max_num_seqs 4`, `gpu 0.6`, `enable_chunked_prefill: false` ("SlowAR decode loop isn't chunked-prefill-safe"), sampling `0.8/top_k 30/top_p 0.9`; stage 1 `enforce_eager: true`, `gpu 0.1`; `platforms.xpu` override (Intel Arc B60). Variants [`_2gpu`](../../vllm-omni/vllm_omni/deploy/fish_qwen3_omni_2gpu.yaml) (64 seqs each GPU), [`_high_concurrency_dual_gpu`](../../vllm-omni/vllm_omni/deploy/fish_qwen3_omni_high_concurrency_dual_gpu.yaml) / [`_single_gpu`](../../vllm-omni/vllm_omni/deploy/fish_qwen3_omni_high_concurrency_single_gpu.yaml) add `fish_speech_tensor_codes`, `fish_speech_single_initial_chunk`, `fish_speech_backlog_codec_chunk_frames: 50`, `fish_speech_dac_dtype: fp16`, `fish_speech_dac_max_padded_frames: 256`. + +**Conditioning** [Verified] — chat-template prompt in [`prompt_utils.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/prompt_utils.py) (`<|im_start|>system … <|voice|>`); voice clone = system prompt with `ref_text` + `<|audio_start|>semantic_ids<|audio_end|>`; no separate instruction field. **DAC encoding of reference audio runs inside stage 0's model on the stage-0 GPU during prefill** (`_build_structured_voice_clone_prefill_embeds` → `encode_reference_audio_codes`, [`dac_encoder.py#L118`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/dac_encoder.py#L118); `torchaudio.transforms.Resample` to 44.1 kHz; the `dac_utils.py` docstring saying "CPU" is stale). Injection: sum of per-codebook embeddings / `sqrt(num_codebooks+1)` at the audio span ([`fish_speech_slow_ar.py#L641`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_slow_ar.py#L641)). Serving fetches audio through vLLM `MediaConnector`; offline example uses `soundfile`. Speaker cache 512 MiB LRU keyed by voice name ([`utils/speaker_cache.py`](../../vllm-omni/vllm_omni/utils/speaker_cache.py)). + +**Generation (dual-AR)** [Verified] — **Slow AR** = vLLM `Qwen3Model` (dim 2560, 36 layers, 32/8 heads, vocab 155776; RoPE rebuilt GPT-J interleaved; logits masked to semantic range `[151678..155773]` + `<|im_end|>`; [`configuration_fish_speech.py#L14`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/configuration_fish_speech.py#L14)). **Fast AR** ([`fish_speech_fast_ar.py#L338`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_fast_ar.py#L338)) = 4-layer / dim-2560 transformer with `max_seq_len=11` predicting 9 residual codebooks (1024 entries) per frame: position 0 = projected slow hidden, position 1 = semantic-code embedding, then a Python loop of 9 single-token steps over a per-call scratch KV `[L,B,kv_heads,11,D]` with plain `F.scaled_dot_product_attention` (**no paged attention**); inline top-k/top-p/multinomial. Invoked through the runner-owned `talker_mtp` hook wrapped in a **full CUDA graph** per batch size when `talker_mtp_graph_safe=True` ([`gpu_ar_model_runner.py#L534`](../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py#L534)); Fast AR for step t uses hidden_{t-1} and code_{t-1} (matches reference flow). Frame rate: hop 2048 @ 44.1 kHz = **21.5 Hz** ([`dac_utils.py#L12`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/dac_utils.py#L12)). + +**DAC decoder** [Verified] — [`fish_speech_dac_decoder.py#L72`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_dac_decoder.py#L72): Descript-style conv+transformer hybrid from the external **`fish-speech` pip package** (`fish_speech.models.dac.modded_dac`; `build_dac_codec` in [`dac_utils.py#L16`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/dac_utils.py#L16): decoder_dim 1536, rates [8,8,4,2], causal, 4 transformer layers in the first decoder block, RVQ 9×1024 + window-limited transformers); weight-norm baked, encoder pruned. Stage 1, eager. Input `[10, frames]` (tensor path) or flat `10*frames` ids; **batched** (`_decode_group` pads `[B,10,T]`, groups bounded by `fish_speech_dac_max_padded_frames`), decode under `autocast(enabled=False)`. Chunked decode has **no cross-chunk conv state**: each chunk is decoded with 25 frames of left overlap then trimmed proportionally (`cut = ctx/total * len(wav)`, L454-470). Output `{"model_outputs": [wav], "sr": [44100]}`. + +**Streaming** [Verified] — `slow_ar_to_dac_decoder_async_chunk` ([`stage_input_processors/fish_speech.py#L117`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/fish_speech.py#L117)): first chunk after `initial_codec_chunk_frames=4` (≈186 ms audio), steady 25-frame chunks with 25 frames of left context, backlog mode grows to 50 frames under load. First audio needs: prefill (+DAC encode for clone) + 4 slow steps (each = Qwen3 decode + 9 fast-AR micro-steps) + SHM poll + one DAC decode of 4(+ctx) frames [Inferred structure]. + +**State** [Verified] — Slow AR KV: vLLM paged attention, optionally with a Triton decode-only kernel ([`attention/fish_kvcache_backend.py`](../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py), env `VLLM_OMNI_FISH_KVCACHE_ATTN`, requires head_dim 128 / block 16). Fast AR: scratch KV reset every step. Cross-step: `hidden_states.last` GPU-resident. Stage 1 stateless. + +**Inter-stage tensors** [Verified] — `OmniPayloadStruct(codes.audio: long [10,F] or flat [10*F], meta.{left_context_size, finished})`; intra-stage `codes.audio [B,10]` per step from `talker_mtp_output_key=("codes","audio")`. + +### 2.4 MOSS-TTS family (`moss_tts_delay` / `moss_tts_realtime` / `moss_tts_local`, plus `moss_tts_nano`) + +**Pipelines** [Verified] — [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/pipeline.py): three 2-stage pipelines (L23, L56, L87) sharing stage 1 `moss_tts_codec` (`LLM_GENERATION`, `MossTTSCodecDecoder`, **`retains_state_across_chunks=True`** — counted by the generation scheduler as parked streaming requests, [`omni_generation_scheduler.py#L172`](../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py#L172)). Delay uses async processor `talker2codec_delay_async_chunk`; Realtime/Local use `talker2codec_raw_async_chunk`. Nano ([`moss_tts_nano/pipeline.py#L19`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano/pipeline.py#L19)) is single-stage (`final_output=True`, `engine_output_type="audio"`), wrapping HF remote code. Deploy variants: [`moss_tts.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts.yaml) (8B, n_vq 32, stage 1 `max_model_len 196608` = 4096 frames × 32 codes + margin, `decode_cudagraph_capture_sizes`), [`moss_tts_realtime.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts_realtime.yaml) (1.7B, n_vq 16, `codec_chunk_frames 15`, claimed "TTFB ~180 ms" — not measured in repo), [`moss_tts_local.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts_local.yaml) (v1.5, n_vq 12, 48 kHz stereo, `initial_codec_chunk_frames: 1`, stage 1 `cudagraph_capture_sizes [1..64]`), plus [`moss_ttsd.yaml`](../../vllm-omni/vllm_omni/deploy/moss_ttsd.yaml), [`moss_sound_effect.yaml`](../../vllm-omni/vllm_omni/deploy/moss_sound_effect.yaml), [`moss_voice_generator.yaml`](../../vllm-omni/vllm_omni/deploy/moss_voice_generator.yaml) (notes the Delay talker is "not CUDA-graph-safe"), [`moss_tts_80gb.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts_80gb.yaml), [`moss_tts_nano.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts_nano.yaml) (0.1B, "verified on 1× L4 24 GB", eager). + +**Conditioning** [Verified] — prompts are built by the upstream HF `AutoProcessor` (`build_user_message(text, reference, instruction, tokens, quality, sound_event, ambient_sound, language)`) into an `(L, 1+n_vq)` grid: column 0 → `prompt_token_ids`, columns 1.. → `additional_information["codes"]["ref"]` ([`end2end.py#L44`](../../vllm-omni/examples/offline_inference/text_to_speech/moss_tts/end2end.py#L44); serving `_build_moss_tts_params` [`serving_speech.py#L1494`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py#L1494)). **Reference audio is encoded outside both stages** (API-server process, `encode_audios_from_wav` on CPU via [`reference_encoder.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/reference_encoder.py) with content-addressed cache, micro-batching 10 ms / 8). The talker adds `codes.ref` embeddings at prefill. Realtime feeds remaining text ids one per decode step via forced one-hot logits (`ids.all`). Loading: `soundfile` + `torchaudio.functional.resample`. + +**Generation** [Verified] — all on vLLM `Qwen3Model` with additive fusion `text_embed + Σ audio_embed_i`. **Delay** ([`modeling_moss_tts_talker.py#L97`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py#L97)): n_vq+1 parallel heads on the final hidden; text head drives vLLM's sampler, audio heads sampled in `make_omni_output` via one stacked matmul with hard-coded `top_k 25 / temp 1.7`; codebook i activates i steps after the delay slot; de-delay in the input processor. **Realtime**: no text head; per-step frame from a 4-layer `MossTTSRealtimeLocalTransformer` re-prefilled over codebooks (no KV) ([`modeling_moss_tts_local.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_local.py)). **Local v1.5**: 1-layer GPT2-style depth transformer ([`modeling_moss_tts_local_depth.py#L35`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_local_depth.py#L35)) with a binary continue/stop head, run through the runner's `talker_mtp` hook **before** the backbone (FULL-graph wrapped when graph-safe), `torch.compile` on the depth prefix. Frame rate **12.5 Hz / 80 ms**; codec sample rate 24 kHz (v1, `downsample_rate 1920`) or 48 kHz stereo channel-interleaved (v2, 3840) ([`audio_tokenizer_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py)). + +**Codec** [Verified] — MOSS Audio Tokenizer: **transformer-only** (patched reshape up/down-sampling + causal RoPE transformer blocks, no strided convs; residual LFQ quantizer 32×1024 default; v2 folds decode projections into a LUT). Stage 1 [`modeling_moss_tts_codec.py#L207`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py#L207): non-streaming path decodes one request at a time (optional B=1 CUDA-graph wrapper [`moss_codec_cudagraph.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/moss_codec_cudagraph.py) per padded-T bucket); streaming path groups requests with identical T into `(NQ,B,T)` batches through `_MossCodecStreamSession` with **per-slot ring KV caches** (`RingKVCache`, capacity = 10 s context) and no overlap (`codec_left_context_frames: 0` everywhere) — graphs keyed `(B_bucket, exact_T)` in [`cuda_graph_streaming_decoder_wrapper.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/cuda_graph_streaming_decoder_wrapper.py). Decoder cast bf16 on non-CPU. + +**Streaming / first audio** [Verified] — **Delay variants do not stream**: `chunk_frames = 1 << 30` in `talker2codec_delay_async_chunk` ([`stage_input_processors/moss_tts.py#L152`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/moss_tts.py#L152)) so codes are emitted only on finish, making `codec_streaming: true / codec_chunk_frames: 25` in five YAMLs inert; stage 1 then splits internally. **Realtime**: raw rows buffered to 15 frames (1.2 s) → first audio after ~15 talker steps + one codec step. **Local**: first codec call after **1 frame** (80 ms), then 15-frame steps. **Nano**: one waveform chunk per upstream `inference_stream()` yield. + +**State** [Verified] — backbone paged KV; per-request `audio_state` dicts + GPU-resident `audio_codes.current/accumulated`, `hidden_states.last`; depth/local transformers stateless; codec ring-KV slots leased per request and released on terminal payload. **Inter-stage**: Delay → `codes.audio` python list (codebook-major); raw → 1-D long `[NQ*T]` + `meta.{req_id, codec_chunk_frames, code_flat_numel, stream_finished, finished}`; talker inputs `codes.ref (L,n_vq)`, `ids.all`, `max_new_frames`. + +### 2.5 TTS cross-family comparison + +| | Qwen3-TTS | CosyVoice3 | Fish S2 Pro | MOSS (Delay / Realtime / Local) | +|---|---|---|---|---| +| Talker backbone | vLLM `Qwen3Model` | vLLM `Qwen2Model` 0.5B | vLLM `Qwen3Model` 36L | vLLM `Qwen3Model` | +| Per-frame sub-AR | 5L code predictor (15 sub-steps, re-prefill) | none (single FSQ token) | 4L fast AR (9 steps, scratch KV) | none / 4L local / 1L depth | +| Codec tokens | 16×2048 @ 12.5 Hz | 6561 @ 25 Hz | 10 cb @ 21.5 Hz | 32/16/12 cb @ 12.5 Hz | +| Vocoder type | conv+sliding-window transformer (fp32) | CFM DiT (10 steps, CFG) + HiFT ISTFT | DAC conv+transformer | transformer-only codec | +| First audio after | 1 frame (80 ms) | ≥18 tokens (~0.7 s) + 10-step CFM | 4 frames (~186 ms) | full utterance / 15 frames / 1 frame | +| Cross-chunk vocoder state | conv ctx 2+12, ref KV, 72-frame window | cumulative mel (re-decode all) | none (25-frame overlap) | ring KV per slot | +| Stage-1 batching | yes (padded, ≤4) | no (sequential loop) | yes (padded groups) | streaming: same-T grouping | +| Stage-1 graphs | stateful CUDA graphs per chunk length | eager (TRT estimator optional) | eager | CUDA graphs `(B,T)` buckets | +| Output SR | 24 kHz | 24 kHz | 44.1 kHz | 24 kHz / 48 kHz stereo | + +### 2.6 Inter-stage tensor contracts (TTS) + +What actually crosses the stage boundary, per family. All wire tensors are serialized to **CPU bytes** by the connector ([`serialization.py#L89`](../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py#L89)) and re-materialized on the consumer's device; the schema is enforced by `OmniPayloadStruct` (`forbid_unknown_fields`) in [`data_entry_keys.py`](../../vllm-omni/vllm_omni/data_entry_keys.py). [Verified] + +| Family | Stage-0 per-step `multimodal_outputs` | Wire payload (async chunk) | Stage-1 consumes as | Stage-1 emits | +|---|---|---|---|---| +| Qwen3-TTS | `codes.audio` long `[span, 16]`, `codes.ref` `[T_ref, 16]`, `meta.ref_code_len`, `meta.codec_streaming`; hidden states **not** shipped | `codes.audio` flat long `[16·F_new]` (codebook-major), `meta.{finished, left_context_size(=0 or ref ctx), ref_context_size, ref_context_request_id, ref_context_included}`, `speaker`, `language` | placeholder `prompt_token_ids` growing per chunk + `runtime_additional_information["codes"]["audio"]` | `{"model_outputs": [float32 [n]], "sr": [int32 24000]}` | +| CosyVoice3 | prefill only: `embed.{speech_token [1,T] int32, speech_feat [1,2T,80], embedding [1,192], speech_token_len}`; `text_hidden_states [tokens, 896]` | `codes.audio` int64 `[prefix_len]` (**cumulative**), `meta.{left_context_size=emitted_len, stream_finished, req_id}`, `embed` on first chunk | flat codec ids as `input_ids` (embedded as zeros `[N,896]`), conditioning via `model_intermediate_buffer` | `{"audio": [float32 1-D], "sr": [24000]}` | +| Fish S2 Pro | `codes.audio` `[B, 10]` per step (`talker_mtp_output_key`), `hidden_states.last [2560]` bf16, `embed.prefill` (CPU pinned) | `codes.audio` long `[10, F]` (tensor mode) or flat `[10·F]`, `meta.{left_context_size=25, finished}` | `[10, F]` tensor or flat ids reshaped codebook-major | `{"model_outputs": [wav], "sr": [44100]}` | +| MOSS Delay | `codes.audio` list of `(T_acc, n_vq)` int64 accumulated snapshots | python list (codebook-major, de-delayed) on **finish only**, `meta.{left_context_size=0, finished}` | flat ids split by `seq_token_counts` → `(n_vq, T)` | `{"model_outputs": [float32 (T_wav,) or (C, T_wav)], "sr"}` | +| MOSS Realtime / Local | `codes.audio` list of `(1, n_vq)` rows | 1-D long `[n_vq·T_chunk]`, `meta.{req_id, codec_chunk_frames, codec_left_context_frames=0, code_flat_numel, stream_finished, finished}` | same, with `retains_state_across_chunks` slot state | same | + +Consequence for an edge port [Inferred]: the only information a codec stage needs from the talker is the integer code stream plus a small metadata record (reference-context length, chunk boundary, finished flag). Speaker conditioning for the vocoder (CosyVoice3's `embed.*`, Qwen3-TTS's `codes.ref`) is sent once at the first chunk, so a single-process edge pipeline can keep it resident and skip the transport entirely. + +### 2.7 First-audio latency decomposition + +Sequential components that must complete before the first PCM sample leaves the pipeline, from the cited code paths (no timings measured here; repo-reported numbers are labelled). [Verified structure / Inferred ordering] + +| Family | 1. Front-end (CPU/GPU) | 2. Talker prefill | 3. Talker decode steps before first chunk | 4. Transport | 5. First vocoder call | Repo-reported | +|---|---|---|---|---|---|---| +| Qwen3-TTS (12 Hz) | tokenizer + text_projection; Base: ECAPA mel + Mimi encoder over ref audio (cached by SHA1) | one vLLM prefill (prompt ≈ instruct + codec tags + text or first text token) | **1 step** = 1 Qwen3 decode + 15 code-predictor sub-prefills (`initial_codec_chunk_frames: 1`; `min_tokens 2` only blocks EOS) | SHM put/get, poll 10 ms, first-chunk wait ≤3 s | stateful first-chunk graph (`_decode_icl_first_chunk` incl. ≤72 ref frames, or xvec path) → 1920 samples (80 ms) | TTFP 64 ms @ c=1, H200 ([doc](../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md)) | +| CosyVoice3 (25 Hz) | s3tokenizer (GPU) + CAM++ (TRT/ORT) + prompt mel; text tokens | prefill of `[SOS, text, TASK, prompt speech tokens]` | `15 + prompt_pad + 3` steps (~0.72 s of speech) | SHM, poll 10 ms | CFM 10 Euler steps × batch 2 (CFG) over prompt+chunk (TRT estimator optional) + HiFT on cumulative mel | none for TTFP; continuity notes only | +| Fish S2 Pro (21.5 Hz) | chat template; clone: DAC **encoder on stage-0 GPU** during prefill | prefill incl. `<\|audio_start\|>…` ref span | **4 steps** (`initial_codec_chunk_frames: 4`, ≈186 ms audio), each = Qwen3 decode + 9 fast-AR micro-steps (CUDA-graphed) | SHM, poll 10 ms (1 ms in HC profiles) | DAC decode of 4 (+ctx) frames, eager, batched | none | +| MOSS Delay (12.5 Hz) | HF `AutoProcessor` grid; ref audio encoded in the API-server process (CPU, cached) | prefill with `codes.ref` added | **entire utterance** (chunking disabled) | one put on finish | full-utterance decode split into 25-frame steps | none | +| MOSS Realtime | grid + `ids.all` text queue | prefill | **15 steps** (1.2 s audio), each with a 4-layer local-transformer re-prefill over 16 codebooks | SHM, poll 5 ms, first wait 1 s | one `(NQ,B,15)` ring-KV decode | "TTFB ~180 ms" (YAML header, unmeasured) | +| MOSS Local v1.5 | grid | prefill | **1 step** (`initial_codec_chunk_frames: 1`), depth transformer runs before the backbone (`talker_mtp`) | SHM, poll 5 ms | `(B,1)` CUDA-graph bucket decode, 48 kHz stereo | none | + +Design reading [Proposal]: for an on-device TTS the components that dominate are (3) and (5). Qwen3-TTS and MOSS-Local show the pattern to copy — emit after a single frame and keep vocoder context resident — while CosyVoice3's flow-matching vocoder needs ≥10 DiT evaluations per chunk and re-decodes history, which is the wrong shape for edge streaming unless the CFM is replaced by an incremental variant. + +--- + +## 3. VLA / world-model deep dive + +Shared request plumbing first (applies to all): `OmniDiffusionRequest` ([`diffusion/request.py#L39`](../../vllm-omni/vllm_omni/diffusion/request.py#L39)) carries `prompt`, `sampling_params: OmniDiffusionSamplingParams`, whose `extra_args: dict[str, Any]` ([`inputs/data.py#L319`](../../vllm-omni/vllm_omni/inputs/data.py#L319)) is **the** pathway for non-text tensors (robot obs, state, noise, camera scripts). Pipelines return `DiffusionOutput(output=...)` ([`diffusion/data.py#L1565`](../../vllm-omni/vllm_omni/diffusion/data.py#L1565)); [`output_formatter.py#L254`](../../vllm-omni/vllm_omni/diffusion/output_formatter.py#L254) copies only keys in `{"audio","actions","trajectory"}` into `OmniRequestOutput.multimodal_output`. Robot policies are served on `WS /v1/realtime/robot/openpi` ([`api_server.py#L1869`](../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py#L1869) → [`entrypoints/openpi/connection.py`](../../vllm-omni/vllm_omni/entrypoints/openpi/connection.py) / [`serving.py`](../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py)): binary msgpack with numpy markers, server first sends `policy_server_config` from the deploy YAML, then request/response `infer` (one action reply per message, no streaming) and `reset` ([`openpi_api.md`](../../vllm-omni/docs/serving/openpi_api.md)); 64 MiB payload cap, 30 s idle timeout. Enabled only when the deploy YAML declares `model_config.policy_server_config`. [Verified] + +### 3.1 GR00T N1.7 (`Gr00tN1d7`) + +- **Pipeline** [Verified]: [`gr00t/pipeline.py#L11`](../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py#L11) single `DIFFUSION` stage, `final_output_type="actions"`, no custom funcs. Deploy [`Gr00tN1d7.yaml`](../../vllm-omni/vllm_omni/deploy/Gr00tN1d7.yaml): `async_chunk: false` (validator requires it for single stage), `max_num_seqs 1`, `enforce_eager: true`, `model_config.embodiment_tag: OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT`, `policy_server_config` (`image_resolution [180,320]`, `action_horizon 40`, `action_keys [eef_9d, gripper_position, joint_position]`). `Gr00tN1d7Pipeline` ([`pipeline_gr00t.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py)) is a thin wrapper: `load_weights` is a no-op; the whole model loads via `AutoModel.from_pretrained(..., bf16)` in [`policy.py#L64`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py#L64). +- **Perception** [Verified]: backbone = stock HF `Qwen3VLForConditionalGeneration` configured from `nvidia/Cosmos-Reason2-2B`, forced `sdpa`, LLM **truncated to 12 layers**, `lm_head` dropped; features = pre-final-norm hidden of the last kept layer `[B, seq, 2048]` via a forward hook ([`gr00t_n1d7.py#L291`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py#L291)). Image preprocessing ([`processing_gr00t_n1d7.py#L49`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/processing_gr00t_n1d7.py#L49)): torchvision v2 `LetterBox → Resize(256) → CenterCrop(230) → Resize(256)` on uint8, then the HF `Qwen3VLProcessor` (from `Qwen/Qwen3-VL-2B-Instruct`) does patching/normalization. Cameras/frames come from checkpoint `modality_configs`; DROID test uses V=2 views × T=2 frames = 4 images/step ([`tests/helpers/client.py#L155`](../../vllm-omni/tests/helpers/client.py#L155)) [Inferred for the count]. +- **Structured inputs** [Verified]: `obs["state"]: {key: float32 (B,T,D)}` (DROID: eef_9d 9 + gripper 1 + joint 7 = 17 raw), min/max-normalized to [-1,1] from checkpoint `statistics.json`, zero-padded to `max_state_dim=132`, projected by a **per-embodiment** `CategorySpecificMLP` (`W[32,132,1024]`→`[32,1024,1536]`, index = embodiment id, DROID→24; [`embodiment_conditioned_mlp.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/modules/embodiment_conditioned_mlp.py)). Language: lower-cased, tokenized by the Qwen3-VL tokenizer (left pad), one chat turn with all images then text. All of it arrives as raw numpy in `extra_args["robot_obs"]` (`video|images`, `state`, `language|prompt`) — **never** `multi_modal_data` ([`pipeline_gr00t.py#L110`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py#L110)). +- **Fusion** [Verified]: vision+language inside the VLM; `vlln` LayerNorm; state fused as one token prepended to 40 action tokens; VL features enter the DiT via cross-attention in `AlternateVLDiT` ([`modules/dit.py#L239`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/modules/dit.py#L239)): even blocks cross-attend (alternating text-only / image-only K/V masks, `attend_text_every_n_blocks=2`), odd blocks self-attend; built on `diffusers.models.attention.Attention` + `AdaLayerNorm`. +- **Action head** [Verified]: `Gr00tN1d7ActionHead` — 16 DiT layers, 32 heads × 48 = 1536, cross-attn dim 2048; `MultiEmbodimentActionEncoder` (per-embodiment W1/W2/W3 with sinusoidal timestep), decoder `CategorySpecificMLP(1024→1024→132)`; flow matching with **4 Euler steps** (`num_inference_timesteps=4`, from model config only — request `num_inference_steps` ignored), horizon **40**, padded action dim 132, no CFG, no scheduler object ([`gr00t_n1d7.py#L123`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py#L123)). RTC inpainting code exists but is unreachable at inference. +- **Temporal state** [Verified]: none — `state_history_length=1`, `reset()` returns `{}`; every request recomputes the full VLM forward (4 images) + 4 DiT steps; frame history must be supplied by the client. +- **Output** [Verified]: `DiffusionOutput(output={"actions": {key: float32 (B,40,D)}})` after un-normalization and relative→absolute conversion using scipy `Rotation` ([`dataio/state_action`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/dataio/state_action)); DROID shapes `eef_9d (1,40,9)`, `gripper_position (1,40,1)`, `joint_position (1,40,7)` ([`test_gr00t_openpi_expansion.py`](../../vllm-omni/tests/e2e/online_serving/test_gr00t_openpi_expansion.py)). +- **Latency per step** [Verified]: CPU transforms + HF processor → H2D → Qwen3-VL prefill (ViT + 12 layers, SDPA, eager) → 4 × (encoder + 16-block DiT + decoder) → D2H → numpy/scipy post. Nothing compiled or graph-captured; startup warm-up returns zeros without touching the model, so the first real request pays cold start. Batch 1; one embodiment per server. +- **Client** [Verified]: no GR00T-specific example; reuse [`examples/online_serving/pi0/openpi_client.py`](../../vllm-omni/examples/online_serving/pi0/openpi_client.py) (`openpi_client.WebsocketClientPolicy`). Serve: `vllm serve nvidia/GR00T-N1.7-3B --omni --port 8000 --served-model-name gr00t-n1d7 --deploy-config vllm_omni/deploy/Gr00tN1d7.yaml` ([recipe](../../vllm-omni/recipes/NVIDIA/GR00T-N1.7.md)); recipe reports ~6 GiB peak bf16 (repo-reported). + +### 3.2 π0 (`pi0`) + +- **Pipeline** [Verified]: [`pi0_pipeline_config.py#L16`](../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py#L16) single `DIFFUSION` stage, `final_output_type="action"`. Deploy [`pi0.yaml`](../../vllm-omni/vllm_omni/deploy/pi0.yaml): `dtype: float32`, `max_num_seqs 1`, `enforce_eager: true` ("flow matching needs no CUDA graph"), `model_config`: `paligemma_variant gemma_2b`, `action_expert_variant gemma_300m`, `chunk_size 50`, `max_action_dim 32`, `max_state_dim 32`, `num_inference_steps 10`, `image_resolution [224,224]`, `tokenizer_max_length 48`, `max_cameras 3`, `policy_server_config`. +- **Model** [Verified]: `PaliGemmaWithActionExpert` = HF `PaliGemmaForConditionalGeneration` (SigLIP + Gemma-2B prefix) + `GemmaForCausalLM` Gemma-300M suffix ([`modeling_pi0.py#L372`](../../vllm-omni/vllm_omni/diffusion/models/pi0/modeling_pi0.py#L372)); hand-rolled eager attention walking Gemma layers manually (prefix-only / suffix-only helpers); flow matching Euler t=1→0, 10 steps, no CFG; prefix = `[3×256 image tokens, 48 text]` bidirectional, suffix = `[state, 50 action-time tokens]`. +- **Inputs** [Verified]: `extra_args["robot_obs"]` — up to 3 cameras in `image_feature_keys` order (missing → −1-filled, masked), `resize_with_pad` to 224 in [-1,1] ([`processor_pi0.py#L48`](../../vllm-omni/vllm_omni/diffusion/models/pi0/processor_pi0.py#L48), numpy/PIL/torch only), `state` zero-padded to 32, `prompt` right-padded to 48 tokens; no embodiment id. Optional `extra_args["noise"]`, `["num_inference_steps"]`. +- **State/Output/Latency** [Verified]: stateless across calls (`session_id`/`reset` ignored); prefix KV computed once per request and reused across the 10 suffix steps only. Output `{"actions": float32 (50,32)}`; un-normalization via `config.norm_stats` (identity for `pi0_base`; README notes LeRobot `policy_preprocessor.json` stats are **not** loaded). Per step: 3× SigLIP + 18-layer Gemma-2B prefix over ~816 tokens + 10 × 18-layer 300M suffix passes; float32, eager, batch 1. +- **Client** [Verified]: [`examples/online_serving/pi0/openpi_client.py`](../../vllm-omni/examples/online_serving/pi0/openpi_client.py) loop `policy.infer(obs)`; serve `vllm serve lerobot/pi0_base --omni --port 8000 --served-model-name pi0 --deploy-config vllm_omni/deploy/pi0.yaml --enforce-eager` ([recipe](../../vllm-omni/recipes/lerobot/Pi0.md)). Parity test runs on **CPU/float32** ([`test_pi0_parity.py`](../../vllm-omni/tests/diffusion/models/pi0/test_pi0_parity.py)). + +### 3.3 DreamZero-DROID (`dreamzero`) + +- **Pipeline** [Verified]: [`dreamzero/pipeline.py#L11`](../../vllm-omni/vllm_omni/model_executor/models/dreamzero/pipeline.py#L11) single `DIFFUSION` stage, `final_output_type="image"` (actions ride in `multimodal_output`). Deploy [`dreamzero.yaml`](../../vllm-omni/vllm_omni/deploy/dreamzero.yaml): `engine_backend: vllm_omni.experimental.ar_diffusion.engine.ARDiffusionEngine` (mandatory, pipeline refuses others), `max_num_seqs 1`, `enforce_eager: false` (but all shipped commands add `--enforce-eager`), `cache_backend: step_cache` (skip DiT steps when velocity cosine-sim > 0.95/0.93), `policy_server_config` (`image_resolution [180,320]`, 2 external + wrist cameras, `needs_session_id`, `action_space joint_position`). [`dreamzero_tp1_cfg2.yaml`](../../vllm-omni/vllm_omni/deploy/dreamzero_tp1_cfg2.yaml) differs only by `devices "0,1"` and `parallel_config {tensor_parallel_size 1, cfg_parallel_size 2}`. +- **Model** [Verified]: `CausalWanModel` ([`causal_wan_model.py`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/causal_wan_model.py), comment L743 "14B: 40 layers, dim 5120, heads 40, ffn 13824"; real dims from checkpoint config) — causal DiT with **action/state tokens appended after video tokens** (own RoPE tables, own `VideoActionScheduler`), `WanI2VCrossAttention` (257 CLIP tokens + 512 UMT5 tokens), fused `QKVParallelLinear` + `paged_write_attn` custom op. Text encoder UMT5-XXL (24L, d 4096, tokenizer `google/umt5-xxl`), image encoder CLIP ViT-H/14 ([`image_encoder.py#L184`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/image_encoder.py#L184)), VAE `DistributedAutoencoderKLWan` (16 ch, 8× spatial, 4 frames/latent). Denoising: `FlowUniPCMultistepScheduler`, **16 steps**, CFG 5.0 (video only; action takes positive branch), sigma shift 5.0 ([`utils.py#L6`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/utils.py#L6)). Action horizon 24, dim 8 (7 joints + gripper, [`transform/droid.py#L42`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/transform/droid.py#L42)), relative joint actions un-normalized with q01/q99 stats from `experiment_cfg/metadata.json`. +- **Inputs** [Verified]: `extra_args["robot_obs"]` with `observation/exterior_image_{0,1}_left`, `observation/wrist_image_left` (`(H,W,3)` first call, `(4,H,W,3)` later), `observation/joint_position (7)`, `observation/gripper_position (1)`, `prompt`, `session_id`; three cameras center-cropped, resized 176×320 and **stitched to 352×640** ([`transform/droid.py#L46`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/transform/droid.py#L46)); state padded to `max_state_dim`; language wrapped in a multi-view template, tokenized to 512. +- **Temporal state** [Verified]: per-session paged KV in the AR-Diffusion runner ([`experimental/ar_diffusion/runner.py`](../../vllm-omni/vllm_omni/experimental/ar_diffusion/runner.py)) with window 21 latent frames, two KV branches (pos/neg CFG), cross-attn caches (text 512, image 257), `session_capacity 64` (LRU), **603 MiB model-owned state per session** (Wan VAE encoder `feat_cache`) ([`pipeline_dreamzero.py#L129`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/pipeline_dreamzero.py#L129)); each request = prefill observation latents into KV → denoise `num_frame_per_block` future latents + action chunk → `current_start_frame += nfpb`; window resets keep prompt/text KV. Optional [`session_state_manager`](../../vllm-omni/docs/features/session_state_manager.md) adapter bounds the VAE history ring. Not `diffusion_kv/`. +- **Output** [Verified]: `{"actions": float32 (24, 8), "video": normalized latents (B,16,T,H/8,W/8) bf16}`; RGB decode only via worker RPC in [`export_prediction_video.py`](../../vllm-omni/examples/offline_inference/dreamzero/export_prediction_video.py). +- **Latency per tick** [Verified]: CPU transform → (first call) UMT5 + CLIP + VAE encode + cross-KV population; (later) incremental VAE encode → 1 DiT prefill pass per CFG branch → 16 × (1–2) DiT passes (step-cache may skip) → denorm. `torch.compile` (`reduce-overhead` encoders, `default` per DiT block) exists but is opt-in since CI/examples force eager. Batch 1, single sequence per runner. +- **Client** [Verified]: [`openpi_client.py`](../../vllm-omni/examples/online_serving/dreamzero/openpi_client.py) (frame schedule offsets `[-23,-16,-8,0]`, cursor +24), [`droid_sim_eval_client.py`](../../vllm-omni/examples/online_serving/dreamzero/droid_sim_eval_client.py) (Isaac Lab; replans every `open_loop_horizon=8` of the 24-action chunk, gripper binarized). Serve: `vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 --served-model-name dreamzero-droid --deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml --enforce-eager --disable-log-stats` ([README](../../vllm-omni/examples/online_serving/dreamzero/README.md)). + +### 3.4 LingBot-World 2.0 (`lingbot_world`) + +- **Pipeline** [Verified]: [`lingbot_world/pipeline.py#L23`](../../vllm-omni/vllm_omni/model_executor/models/lingbot_world/pipeline.py#L23) single `DIFFUSION` stage → `video`; deploy [`lingbot_world_v2_stepwise.yaml`](../../vllm-omni/vllm_omni/deploy/lingbot_world_v2_stepwise.yaml): `ARDiffusionEngine`, `step_execution: true`, `streaming_output: true`, `num_inference_steps 4`, `ar_diffusion_height/width 480×832`, KV `gpu_memory_fraction 0.6`. +- **Model** [Verified]: `CausalLingBotWorldTransformer3DModel` ([`transformer.py#L675`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/transformer.py#L675)): Wan-style 40-layer / dim 5120 DiT, block-causal self-attn with **sink 9 + sliding window 18 latent frames**, `num_frames_per_block 3`, camera AdaLN injection from Plücker embeddings ([`camera.py`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/camera.py)); UMT5 text (512), `DistributedAutoencoderKLWan`, DMD 4 timesteps `(1000,750,500,250)` with 4 probe + 1 commit forwards per block ([`dmd_block.py`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/dmd_block.py)); no CFG. +- **Controls** [Verified]: `prompt.multi_modal_data.image` (first frame, required) + `extra_args`: `action_path` (poses/intrinsics npy under a trusted root), `camera_action_script` (per-chunk WASD/IJKL lists, integrated in [`actions.py#L274`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/actions.py#L274)), or a typed `ar_diffusion_tick`. Served via `WS /v1/realtime/video` ([`streaming_video_output_api.md`](../../vllm-omni/docs/serving/streaming_video_output_api.md)) emitting one m4s chunk per AR block; mid-session camera interaction not wired ([recipe](../../vllm-omni/recipes/Robbyant/LingBot-World-2.0.md)). This is a world model without an action head — relevant to VLA only as the KV/session template. No AMD/NPU/XPU support claimed. + +### 3.5 InternVLA-A1 (`internvla_a1`) + +[Verified] No `PipelineConfig`, no deploy YAML, not in `OMNI_PIPELINES`; **offline-only** via `initialize_model(OmniDiffusionConfig(model_class_name="InternVLAA1Pipeline"))` and direct `pipeline.forward` ([`examples/offline_inference/internvla_a1/end2end.py`](../../vllm-omni/examples/offline_inference/internvla_a1/end2end.py)). Model ([`model_internvla_a1.py`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py)): three Qwen3-VL experts (understanding `Qwen3VLForConditionalGeneration` 28L/2048, generation and action `Qwen3VLTextModel` 28L/1024) with joint attention (`compute_layer_complete` concatenates Q/K/V across experts), plus a vendored NVIDIA Cosmos CI8x8 continuous image tokenizer ([`cosmos_ci_torch.py`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/cosmos_ci_torch.py)) whose latents form a "middle" token stream; flow-matching action head, chunk 50, dim 32 (16 used), 10 steps, no CFG; inputs `extra_args["batch_inputs"]` with 3 cameras × T=2 frames `[B,2,3,H,W]` in [0,1], `observation.state` (pre-normalized by the example), `observation.task`. Optional `torch.compile(max-autotune)` / `regionally_compile`; bf16 eager; doc reports 38 s one-sample e2e on H200 ([`internvla_a1.md`](../../vllm-omni/docs/user_guide/examples/offline_inference/internvla_a1.md), repo-reported). `supported_models.md` lists online ✅ but no serving path exists in source [Verified]; `end2end.py` passes `prompts=` to a dataclass whose field is `prompt` [Inferred possible bug, not executed]. + +### 3.6 Cosmos3 action policy (brief) + +[Verified] [`diffusion/models/cosmos3/action.py`](../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py): modes `policy | forward_dynamics | inverse_dynamics`, `EMBODIMENT_TO_DOMAIN_ID`; RoboLab defaults chunk 32, raw action dim 8, 540×640, 4 steps, guidance 3.0; requires external `cosmos_framework` for the action transform. Not analysed further. + +### 3.7 VLA cross-family comparison + +| | GR00T N1.7 | π0 | DreamZero | InternVLA-A1 | +|---|---|---|---|---| +| Vision/LM backbone | Qwen3-VL (Cosmos-Reason2-2B, 12/… layers), HF classes | PaliGemma (SigLIP + Gemma-2B), HF classes | CLIP ViT-H + UMT5-XXL + Wan VAE, reimplemented | Qwen3-VL ×3 experts + Cosmos CI8x8 | +| Images / step | V×T from checkpoint (DROID 2×2), 256² | 3 × 224² | 3 cams stitched 352×640, 4-frame history | 3 × 224² × 2 frames | +| State entry | `robot_obs["state"]` dict, per-embodiment MLP | `robot_obs["state"]` → 32-pad | `robot_obs` joint+gripper → 64-pad | `batch_inputs["observation.state"]` | +| Action head | 16-block DiT cross-attn, flow, 4 steps, H=40 | Gemma-300M expert, flow, 10 steps, H=50 | actions as DiT tokens, 16 UniPC steps, CFG, H=24 | act expert, flow, 10 steps, H=50 | +| Cross-step state | none | none | paged KV per session + VAE stream | none | +| Compile/graphs | none (eager) | none (fp32 eager) | opt-in compile; eager in CI | opt-in compile | +| Serving | OpenPI WS | OpenPI WS | OpenPI WS (session) | none (offline) | + +### 3.8 Control-loop request/response contracts + +Exact observation and action shapes as consumed/produced in tree (batch dimension included where the code requires it). [Verified from the cited files] + +| Family | Observation keys (in `extra_args["robot_obs"]` / `batch_inputs`) | Shapes / dtypes | Reply | Session semantics | +|---|---|---|---|---| +| GR00T N1.7 | `video` or `images`: `{cam: uint8 (B,T,H,W,3)}`; `state`: `{key: float32 (B,T,D)}`; `language`: `{key: [[str]]}` or `prompt`; ([`pipeline_gr00t.py#L135`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py#L135)) | DROID: 2 cams × T=2 × 256² uint8; `eef_9d (1,1,9)`, `gripper_position (1,1,1)`, `joint_position (1,1,7)`; any image size accepted (letterboxed to 256²) | `{key: float32 (B,40,D)}` — `eef_9d (1,40,9)`, `gripper_position (1,40,1)`, `joint_position (1,40,7)`, un-normalized, relative→absolute applied | stateless; `reset` is a no-op; `policy_server_config.needs_session_id: true` only for client bookkeeping | +| π0 | `observation.images.*` / `images` (≤3 cams, `image_feature_keys` order), `state`, `prompt`; ([`processor_pi0.py#L162`](../../vllm-omni/vllm_omni/diffusion/models/pi0/processor_pi0.py#L162)) | HWC uint8 224² (resized with pad), `state` float32 ≤32, prompt ≤48 tokens | `actions` float32 `(50, 32)` (batch squeezed), identity norm for `pi0_base` | stateless; `session_id`/`reset` ignored | +| DreamZero | `observation/exterior_image_{0,1}_left`, `observation/wrist_image_left`, `observation/joint_position`, `observation/gripper_position`, `prompt`, `session_id`, optional `embodiment`; ([`pipeline_dreamzero.py#L1361`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/pipeline_dreamzero.py#L1361)) | images `(H,W,3)` on first call then `(4,H,W,3)` history (180×320 from clients), joint `(7)`, gripper `(1)` | `actions` float32 `(24, 8)`; `video` = latents `(B,16,T,H/8,W/8)` in `images` | **session-affine**: paged KV + VAE stream per `session_id`, LRU 64; `reset` on first call / explicit / prompt change; `extra_args["close_session"]` | +| InternVLA-A1 | `batch_inputs`: `observation.images.image{0,1,2}` `[B,2,3,H,W]` float [0,1] + `_mask`, `observation.state` (pre-normalized, 16→32 pad), `observation.task` str; `noise`, `decode_image`; ([`pipeline_internvla_a1.py#L235`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/pipeline_internvla_a1.py#L235)) | B=1 enforced; history T=2 (offsets −15, 0) | `payload.actions` `(1, 50, 32)` (first 16 dims meaningful), optional decoded image in `metadata` | stateless; offline `pipeline.forward` only | +| LingBot-World (for reference) | `multi_modal_data.image` + `extra_args.{action_path \| camera_action_script \| ar_diffusion_tick, flow_shift}` | first frame 480×832; per-chunk 3-frame WASD lists | video chunk (m4s) per AR block, `metadata.ar_diffusion.{session_id, chunk_index, applied_event_ids}` | session KV (sink 9 + window 9 latent frames), `session_capacity 2` | + +Latency-relevant client behaviour [Verified]: the DreamZero sim client replans every 8 executed actions of a 24-action chunk ([`droid_sim_eval_client.py`](../../vllm-omni/examples/online_serving/dreamzero/droid_sim_eval_client.py)); the π0/GR00T clients issue one `infer` per chunk with no overlap; no client in tree pipelines the next observation while the previous chunk executes (asynchronous inference would need server-side session state that only DreamZero has) [Inferred]. + +--- + +## 4. Cross-cutting pipeline mechanics + +```mermaid +graph LR + C[Client: /v1/audio/speech, /v1/realtime/robot/openpi, Omni.generate] --> E[AsyncOmniEngine] + E --> O[Orchestrator: request state, routing, abort] + O --> S0[Stage 0 process: LLM_AR (vLLM engine + OmniARScheduler)] + S0 -- OmniConnector put/get (SHM, msgpack, CPU tensors) --> S1[Stage 1 process: LLM_GENERATION (GPUGenerationModelRunner) or DIFFUSION worker] + S1 --> MO[OmniRequestOutput.multimodal_output / streaming chunks] + MO --> E +``` + +- **Stage config format** [Verified]: `PipelineConfig(model_type, model_arch, hf_architectures, stages=(StagePipelineConfig(...),), default_deploy_config_name, endpoint_restrictions, duplex_*)` and `StagePipelineConfig(stage_id, model_stage, execution_type ∈ {LLM_AR, LLM_GENERATION, DIFFUSION}, input_sources, final_output/_type, owns_tokenizer, engine_output_type, model_arch, retains_state_across_chunks, sampling_constraints, custom_process_input_func, custom_process_next_stage_input_func, async_chunk_process_next_stage_input_func, sync_process_input_func, prompt_transform_func, prompt_expand_func, cfg_kv_collect_func, omni_kv_config, scheduler_cls, model_subdir, tokenizer_subdir, inline_diffusion, requires_full_payload_input, extras)` ([`stage_config.py#L229`](../../vllm-omni/vllm_omni/config/stage_config.py#L229)). Deploy YAML schema = `DeployConfig` (`async_chunk`, `session_mode`, `connectors`, `edges`, `stages: [StageDeployConfig]`, `platforms: {npu|rocm|xpu: {stages}}`, pipeline-wide dtype/quantization/TP/PP) with per-stage `StageDeployConfig` (devices, num_replicas, connectors, sampling defaults, engine sizing, ~60 diffusion knobs, `engine_extras` catch-all) ([`#L371`](../../vllm-omni/vllm_omni/config/stage_config.py#L371); doc [`stage_configs.md`](../../vllm-omni/docs/configuration/stage_configs.md)). Precedence: `--stage-overrides` > global CLI > platform section > `base_config` overlay > defaults. Scheduler per exec type: `OmniARScheduler`/`OmniARAsyncScheduler`, `OmniGenerationScheduler`, `None` for diffusion ([`#L202`](../../vllm-omni/vllm_omni/config/stage_config.py#L202)). Single-stage diffusion models without a registered pipeline use `_create_default_diffusion_stage_cfg` ([`async_omni_engine.py#L1013`](../../vllm-omni/vllm_omni/engine/async_omni_engine.py#L1013)). +- **Stage output → next-stage input** [Verified]: producer side, `custom_process_next_stage_input_func` (full payload) or `async_chunk_process_next_stage_input_func` (per step) is invoked by the transfer adapter in the main thread, then `connector.put` in a background `save_loop` ([`chunk_transfer_adapter.py#L654`](../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py#L654); design [`async_chunk.md`](../../vllm-omni/docs/design/feature/async_chunk.md)); consumer side, `custom_process_input_func` / `sync_process_input_func` run in the stage client / orchestrator ([`orchestrator.py#L2548`](../../vllm-omni/vllm_omni/engine/orchestrator.py#L2548), [`stage_engine_core_client.py#L398`](../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py#L398)); async-selection logic in `_select_processor_funcs` ([`stage_config.py#L875`](../../vllm-omni/vllm_omni/config/stage_config.py#L875)). Payload schema = `OmniPayload` TypedDicts / msgspec `OmniPayloadStruct` (`hidden_states`, `embed`, `ids`, `codes.{audio,ref}`, `meta.*`, `latent`, `speaker`, `language`) validated with `forbid_unknown_fields` ([`data_entry_keys.py`](../../vllm-omni/vllm_omni/data_entry_keys.py)). No `action`/`video` keys exist in `OmniPayload` — VLA/diffusion models do not use it. +- **Connectors** [Verified]: registered names `SharedMemoryConnector`, `MooncakeStoreConnector`, `MooncakeTransferEngineConnector`, `MoriTransferEngineConnector`, `YuanrongConnector`, `YuanrongTransferEngineConnector` ([`factory.py#L129`](../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py#L129)). SHM connector writes POSIX shared memory under `/dev/shm` with a flock, serializing via `OmniSerializer` (msgpack; **tensors moved to CPU and copied as bytes**, [`serialization.py#L89`](../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py#L89); `data_entry_keys._serialize_tensor` also does `.to("cpu")`). Hence every stage hop is a D2H → IPC → H2D round trip [Inferred consequence]. Same-host only; Mooncake for cross-host. +- **Streaming output paths** [Verified]: AR stage emits per decode step; generation stage batches chunk payloads; HTTP `/v1/audio/speech` returns wav/mp3/flac/pcm/opus via `soundfile` ([`audio_utils_mixin.py#L106`](../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py#L106), resampling with `torchaudio.functional.resample`), streams raw bytes (`stream_format="audio"`) or OpenAI `speech.audio.delta` SSE; WebSocket `/v1/audio/speech/stream` accepts incremental text and returns per-sentence PCM chunks with optional word timestamps ([`serving_speech_stream.py`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py); [`speech_api.md`](../../vllm-omni/docs/serving/speech_api.md)). Per-model request normalisation/validation lives in [`tts_adapters/`](../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters) (one adapter per TTS family, `ARTTSAdapter`/`DiffusionTTSAdapter`). Robot actions: one msgpack reply per request, no streaming. +- **Output objects** [Verified]: `OmniRequestOutput(RequestOutput)` ([`outputs/__init__.py#L108`](../../vllm-omni/vllm_omni/outputs/__init__.py#L108)) adds `stage_id`, `replica_id`, `final_output_type` ("text"/"image"/"audio"/"latents"), `images`, `latents`, `multimodal_output` and plain-dict `metrics` to vLLM's `RequestOutput`; TTS audio arrives as `multimodal_output["audio"]`/`["model_outputs"]` + `"sr"`, robot actions as `multimodal_output["actions"]`, DreamZero video latents in `images`. Diffusion-side, `OmniRequestOutput.from_diffusion` is built by [`output_formatter.py`](../../vllm-omni/vllm_omni/diffusion/output_formatter.py) which unwraps `{"payload", "metadata"}` envelopes. +- **Deploy precedence and platform overrides** [Verified]: `--stage-overrides '{"1": {...}}'` > explicit global CLI flags > `platforms.{npu|rocm|xpu}.stages` > `base_config` overlay > bundled YAML > parser defaults ([`stage_configs.md`](../../vllm-omni/docs/configuration/stage_configs.md)); the stage-based CLI (`--stage-id N --headless --omni-master-address/port`) runs one stage per process. TTS YAMLs already carry per-platform escape hatches (Qwen3-TTS: ROCm stage-1 eager, NPU piecewise graphs; Fish: XPU memory/len limits) — evidence that vocoder graph capture is the first thing to break off-CUDA [Inferred]. +- **Abort propagation** [Verified]: client disconnect → `serving_speech_stream` calls `engine_client.abort(request_id)` ([`#L365`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py#L365)) → `AsyncOmni._abort` ([`async_omni.py#L1181`](../../vllm-omni/vllm_omni/entrypoints/async_omni.py#L1181)) → `AsyncOmniEngine.abort_async` (RPC with ack, [`async_omni_engine.py#L1931`](../../vllm-omni/vllm_omni/engine/async_omni_engine.py#L1931)) → `Orchestrator._handle_abort` / `_abort_request_ids` → each stage pool's `abort_requests` ([`orchestrator.py#L897`](../../vllm-omni/vllm_omni/engine/orchestrator.py#L897)); partial AR tokens returned as terminal abort outputs. Model-level per-request cleanup hooks (`on_requests_finished`) release codec state slots (MOSS) / decoder caches (Qwen3-TTS). +- **Device/dtype assumptions** [Verified]: platforms `cuda`, `rocm`, `npu` (Ascend, incl. 310P patches), `xpu`, `musa` selected by probing pynvml/amdsmi/torch.npu ([`platforms/__init__.py`](../../vllm-omni/vllm_omni/platforms/__init__.py)); **no CPU platform**. Only compiled source: `diffusion/distributed/csrc/a2a_permute.cu` JIT-built via `torch.utils.cpp_extension.load` for Ulysses all-to-all ([`a2a_permute.py#L64`](../../vllm-omni/vllm_omni/diffusion/distributed/a2a_permute.py#L64)) — not used by TTS/VLA. `setup.py` compiles nothing; it only routes requirements by detected device ([`setup.py#L43`](../../vllm-omni/setup.py#L43)). Talkers are bf16 (vLLM default); vocoders mostly fp32 (Qwen3-TTS, HiFT, NeMo RVQ-VAE) or bf16 (MOSS codec on GPU); π0 float32. +- **Dependencies** [Verified]: [`requirements/common.txt`](../../vllm-omni/requirements/common.txt) — `transformers>=5.10.1,<5.15`, `diffusers==0.40.0`, `accelerate`, `soundfile>=0.13.1`, `openai-whisper`, `x-transformers`, `einops`, `cache-dit`, `torchsde`, `gguf`, `cosmos-guardrail`, `msgpack`, `pyzmq`; `cuda.txt` adds `onnxruntime`, `fa3-fwd`; `npu.txt` pins `torchaudio==2.10.0`; `xpu.txt` `auto-round-lib`; `musa.txt` `flash_attn_3`. vLLM itself is **not** in requirements — docs pin `vllm==0.28.0` ([`cuda.inc.md#L23`](../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md#L23)). Optional extras in [`pyproject.toml`](../../vllm-omni/pyproject.toml): `forced-aligner` (`qwen-asr`), `indextts2` (fugashi/WeTextProcessing), `fa4`, `quack`; dev extras include `step-audio2`, `voxcpm`, `s3tokenizer`, `funasr`, `mooncake-transfer-engine-cuda13`. Family-specific external packages: `fish-speech` (DAC), `tensorrt` (CosyVoice3/Step-Audio2), NeMo (Gepard/IndexTTS2), `openpi-client` (VLA clients), `cosmos_framework` (Cosmos3). +- **Session/duplex state** [Verified]: turn-based by default; `session_mode: "duplex"` + `DuplexSessionRuntimeConfig` for MiniCPM-o/PersonaPlex/Nemotron VoiceChat ([`stage_config.py#L499`](../../vllm-omni/vllm_omni/config/stage_config.py#L499), [`fullduplex.md`](../../vllm-omni/docs/design/fullduplex.md)); AR-diffusion world models keep per-session paged KV and model-owned state under [`experimental/ar_diffusion`](../../vllm-omni/vllm_omni/experimental/ar_diffusion) / [`experimental/world_models`](../../vllm-omni/vllm_omni/experimental/world_models) ([`realtime_ar_diffusion.md`](../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md)). + +```mermaid +sequenceDiagram + participant C as Client (/v1/audio/speech stream) + participant S0 as Stage0 Talker (LLM_AR) + participant X as SHM connector + participant S1 as Stage1 Code2Wav (LLM_GENERATION) + C->>S0: prompt embeds (text + ref codes/speaker) + S0->>S0: prefill (paged KV) + loop each decode step + S0->>S0: sample codebook-0, code predictor -> Q codes + S0->>X: async_chunk func: put new frames when chunk boundary + end + X-->>S1: get chunk (CPU tensor), append to growing request + S1->>S1: stateful decode (conv ctx + window KV), batched across requests + S1-->>C: PCM chunk (first after 1 frame for Qwen3-TTS) + S0->>X: finished=True + S1-->>C: last chunk, release state +``` + +--- + +## 5. Portability assessment per family + +Legend: **A** plain torch / HF (exportable to ONNX/TorchScript/ExecuTorch in principle), **B** depends on vLLM layer classes or paged-attention runtime (needs re-implementation for an edge runtime), **C** CUDA-only kernels/graphs/TRT (must be dropped or replaced). Parameter counts are labels (source in §1). + +| Family | Sub-module | Class | Notes | +|---|---|---|---| +| Qwen3-TTS | 12 Hz tokenizer decoder ([`modeling_qwen3_tts_tokenizer_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py)) | A | torch + transformers mask utils/`DynamicCache`; SnakeBeta Triton has torch fallback; NPU patches RoPE/RMSNorm. Causal convs + 72-frame window make it a clean streaming candidate [Inferred]. | +| | Talker (`Qwen3TTSTalkerForConditionalGeneration`) | B | vLLM `Qwen3Model`, `ParallelLMHead`, `LogitsProcessor`; ECAPA speaker encoder is plain nn. | +| | Code predictor | A/B | math is `nn.Linear` + SDPA (design note in file); wrapped by `VocabParallelEmbedding`/`CustomOp` and `torch.compile` — bypassable [Inferred]. | +| | Graph wrappers, forced aligner | C | `torch.cuda.CUDAGraph`; drop on edge. | +| CosyVoice3 | CFM ([`cfm.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py)), HiFT ([`hifigan.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py)), layers | A | plain torch (+omegaconf); HiFT has data-dependent `finalize` branches and a 300 s sine buffer that complicate static export [Inferred]. | +| | DiT estimator ([`cosyvoice3_dit.py`](../../vllm-omni/vllm_omni/diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py)) | A/B | standard math but uses `vllm_omni.diffusion.attention.layer.Attention`, diffusers `AdaLayerNormZero`, `x_transformers` rotary — swap attention for SDPA [Proposal]. An fp16 ONNX export already exists upstream (`yuekai/Fun-CosyVoice3-0.5B-2512-FP16-ONNX`) [Verified reference]. | +| | Talker (`Qwen2Model`), s3tokenizer, CAM++ | B / A | talker on vLLM; CAM++ is ONNX (onnxruntime) — portable; s3tokenizer is a GPU pip package. | +| | TRT paths | C | optional; `COSYVOICE3_TRT=0` disables. | +| Fish S2 Pro | DAC decoder/encoder | A (external) | needs `fish-speech` pip package; plain torch inside; decoder uses `torch.amp.autocast("cuda")` literal (harmless off-CUDA) [Inferred]. | +| | Slow AR | B (+C optional) | vLLM `Qwen3Model`; optional Triton KV backend. | +| | Fast AR ([`fish_speech_fast_ar.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_fast_ar.py)) | A/B | SDPA + scratch KV, but layers are vLLM `Linear/RMSNorm/RoPE (disable_tp)`; small (4 layers). | +| MOSS-TTS | Codec ([`audio_tokenizer_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py)) | A | transformers `PreTrainedModel` + SDPA, ring KV; no convs → attractive for NPU/GPU-less export [Inferred]. | +| | Talkers, depth/local transformers | B | vLLM `Qwen3Model`; depth transformer itself is plain torch (1–4 layers). | +| | Graph wrappers | C | CUDA-only, gated by `device.type == "cuda"`. | +| | Nano | A (HF remote code) | not vLLM-native at all; 0.1B; `flash_attn` optional else sdpa. | +| Gepard / VoxCPM2 / dots.tts / Ming-TTS | codecs | A / external | NeMo NanoCodec (Gepard) is an external dep; AudioVAE (Ming/dots/VoxCPM2) plain torch; CAM++ via onnxruntime. | +| GR00T N1.7 | `policy.py`, `modeling/`, `dataio/`, `configs/` | **A** | torch + transformers (Qwen3-VL) + diffusers (`Attention`, `FeedForward`, `AdaLayerNorm`) + torchvision + scipy; no vLLM kernels; only `pipeline_gr00t.py` and two `vllm.logger` imports bind it to vLLM-Omni. Needs HF hub access for Cosmos-Reason2-2B config and Qwen3-VL-2B processor unless cached [Inferred]. | +| π0 | `modeling_pi0.py`, `processor_pi0.py` | **A** | torch + transformers (Gemma/PaliGemma); vLLM import only for `default_weight_loader`; parity test runs on CPU/fp32. | +| InternVLA-A1 | model + Cosmos tokenizer | A | torch/einops/transformers; `regionally_compile` optional. | +| DreamZero | everything | B/C | vLLM TP layers, custom `paged_write_attn` (flash-attn on CUDA, aiter on ROCm, torch reference elsewhere), AR-Diffusion engine; 14B — not an edge candidate. | +| LingBot-World | transformer | B/C | vLLM parallel layers, paged attention; 14B. | +| Shared | `common/ops/*`, `snake_activation.py` | C with fallback | Triton kernels guarded by `HAS_TRITON` / `is_cuda`. | + +Suggested extraction boundaries per family [Proposal] — "keep" = reuse the in-tree module as the reference implementation for export; "re-implement" = rewrite against the edge runtime's primitives; "drop" = server-only. + +| Family | Keep (export as-is) | Re-implement on edge runtime | Drop | +|---|---|---|---| +| Qwen3-TTS | 12 Hz decoder (`Qwen3TTSTokenizerV2Decoder`, incl. its explicit conv/KV contexts), ECAPA speaker encoder, prompt-template logic from `prompt_embeds_builder.py` | Qwen3 talker decode loop with extra embedding tables + codebook-0 head; code predictor (5-layer, 15 sub-prefills/frame) | CUDA-graph wrappers, SHM transport, forced aligner, 25 Hz tokenizer | +| MOSS-TTS-Realtime / Local | transformer codec with ring KV (`audio_tokenizer_v2.py`), depth/local transformer | Qwen3 backbone loop with additive multi-codebook embeddings | delay-pattern variants (8B, non-streaming), graph wrappers, reference-encoder service | +| CosyVoice3 | HiFT vocoder, CAM++ ONNX, text tokenizer | Qwen2-0.5B talker + RAS sampler; CFM DiT with SDPA and an **incremental** (non-cumulative) chunk schedule | s3tokenizer GPU dependency (needs an ONNX speech tokenizer), TRT paths | +| Fish S2 Pro | DAC decoder/encoder (via `fish-speech`), fast AR (4 layers) | Qwen3-36L slow AR (largest talker of the four — likely too big) | Triton KV backend, graphs | +| GR00T N1.7 | entire `policy.py` + `modeling/` + `dataio/` (HF/torch/diffusers) | nothing mandatory; optionally cache VLM features across steps when images are unchanged | vLLM-Omni pipeline wrapper, OpenPI server (replace with in-process loop) | +| π0 | `modeling_pi0.py` + `processor_pi0.py` | prefix KV caching across steps (not present in tree) | same as above | +| DreamZero / LingBot-World | — | — | out of edge scope (14B, session paged KV, CFG parallel) | + +For an edge TTS port the minimal extraction is: (1) talker weights → any small-LLM runtime with KV cache (the talker is a stock Qwen2/Qwen3 decoder plus extra embedding tables and heads); (2) the per-frame sub-AR (code predictor / fast AR / depth transformer) as a tiny stateless transformer re-prefilled per codebook (≤5 layers, 2–16 sub-steps per frame); (3) the codec decoder as a standalone streaming torch module with explicit context state (Qwen3-TTS's 2/12-frame conv contexts + windowed KV, or MOSS's ring KV, are already factored). For VLA, GR00T and π0 are directly liftable as HF/torch graphs; the serving contract to keep is the OpenPI msgpack `infer/reset` protocol with `policy_server_config` handshake. + +--- + +## 6. Tests and examples + +- **Layout** [Verified]: [`tests/`](../../vllm-omni/tests) — `model_executor/models//` CPU unit tests (`core_model`, `cpu` marks), `model_executor/stage_input_processors/`, `diffusion/models//`, `e2e/offline_inference/`, `e2e/online_serving/`, `e2e/accuracy/`, `entrypoints/openai_api/`, `attention/`, `platforms/`, `dfx/` (perf/stability). Markers in [`pyproject.toml`](../../vllm-omni/pyproject.toml): CI levels `core_model` (L1/L2 per PR), `advanced_model` (L3 merge), `full_model` (L4 nightly), `local_model`; module marks `tts`, `diffusion`, `omni`; hardware marks `cpu/cuda/rocm/xpu/npu/musa` + SKU (`H100`, `L4`, `MI325`, `B60`, `A2/A3`, `S5000`) + `cards_N`; a pre-commit hook enforces one level + one platform mark per test file ([`docs/contributing/README.md`](../../vllm-omni/docs/contributing/README.md)). Buildkite lanes: [`.buildkite/cuda`](../../vllm-omni/.buildkite/cuda) (ready/merge/nightly/weekly), [`amd`](../../vllm-omni/.buildkite/amd), [`npu`](../../vllm-omni/.buildkite/npu), [`intel`](../../vllm-omni/.buildkite/intel). Run per docs: `pytest tests/...` with markers; L1 CPU tests need no weights. +- **TTS tests** [Verified]: Qwen3-TTS — [`tests/model_executor/models/qwen3_tts/`](../../vllm-omni/tests/model_executor/models/qwen3_tts) (incremental decode, CUDA-graph decoder equivalence, talker preprocess/prompt len/ref codes/silence ban, custom voice profiles), [`test_qwen3_tts_async_chunk.py`](../../vllm-omni/tests/model_executor/stage_input_processors/test_qwen3_tts_async_chunk.py), e2e [`test_qwen3_tts_base.py`](../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_base.py) / [`test_qwen3_tts_customvoice.py`](../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_customvoice.py) and online counterparts. CosyVoice3 — [`tests/model_executor/models/cosyvoice3/`](../../vllm-omni/tests/model_executor/models/cosyvoice3) (components incl. TRT stream deps, helpers, utils, config), [`test_cosyvoice3_stage_input_processors.py`](../../vllm-omni/tests/model_executor/stage_input_processors/test_cosyvoice3_stage_input_processors.py), e2e [`test_cosyvoice3_expansion.py`](../../vllm-omni/tests/e2e/offline_inference/test_cosyvoice3_expansion.py) (L4; sync + async params) / [`online`](../../vllm-omni/tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py) (H100/MI325, `COSYVOICE3_TRT=0` on ROCm). Fish — [`tests/model_executor/models/fish_speech/`](../../vllm-omni/tests/model_executor/models/fish_speech), [`test_fish_speech_async_chunk.py`](../../vllm-omni/tests/model_executor/stage_input_processors/test_fish_speech_async_chunk.py), [`tests/attention/test_fish_kvcache_attn.py`](../../vllm-omni/tests/attention/test_fish_kvcache_attn.py). MOSS — [`tests/model_executor/models/moss_tts/`](../../vllm-omni/tests/model_executor/models/moss_tts) (codec CUDA graph, compact top-k, depth RoPE cache, fused load, per-request codes, reference encoder, logits processor); e2e [`test_moss_tts_expansion.py`](../../vllm-omni/tests/e2e/offline_inference/test_moss_tts_expansion.py), [`test_moss_tts_realtime.py`](../../vllm-omni/tests/e2e/offline_inference/test_moss_tts_realtime.py), [`test_moss_tts_nano_expansion.py`](../../vllm-omni/tests/e2e/offline_inference/test_moss_tts_nano_expansion.py) — several MOSS e2e tests are **skipped** (issues #4700, #6417). TTS API/adapters: [`test_serving_speech.py`](../../vllm-omni/tests/entrypoints/openai_api/test_serving_speech.py), [`test_tts_adapter.py`](../../vllm-omni/tests/entrypoints/openai_api/test_tts_adapter.py). Benchmarks: [`benchmarks/tts/`](../../vllm-omni/benchmarks/tts) (`vllm bench serve --omni --backend openai-audio-speech --endpoint /v1/audio/speech --percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,...`; registry [`model_configs.yaml`](../../vllm-omni/benchmarks/tts/model_configs.yaml)). +- **VLA tests** [Verified]: GR00T — [`tests/diffusion/models/gr00t/test_pipeline.py`](../../vllm-omni/tests/diffusion/models/gr00t/test_pipeline.py) (CPU, fake policy), e2e [`test_gr00t_openpi_expansion.py`](../../vllm-omni/tests/e2e/online_serving/test_gr00t_openpi_expansion.py) (L4; precision vs Isaac-GR00T reference `atol=1e-2`, `GR00T_NOISE_SEED=42`). π0 — [`test_pi0_units.py`](../../vllm-omni/tests/diffusion/models/pi0/test_pi0_units.py) (CPU), [`test_pi0_parity.py`](../../vllm-omni/tests/diffusion/models/pi0/test_pi0_parity.py) (vs LeRobot, `local_model`, atol 1e-4), e2e [`test_pi0_expansion.py`](../../vllm-omni/tests/e2e/online_serving/test_pi0_expansion.py) (H100, `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA`). DreamZero — [`tests/diffusion/models/dreamzero/`](../../vllm-omni/tests/diffusion/models/dreamzero) (deploy configs, session state accounting/equivalence/lifecycle, fused QK norm), e2e [`test_dreamzero_expansion.py`](../../vllm-omni/tests/e2e/online_serving/test_dreamzero_expansion.py) (H100×2, expects 3 × `(24,8)`), accuracy [`test_dreamzero.py`](../../vllm-omni/tests/e2e/accuracy/test_dreamzero.py) (upstream parity, `rtol 1e-2 / atol 1e-3`). LingBot-World — [`tests/diffusion/models/lingbot_world/`](../../vllm-omni/tests/diffusion/models/lingbot_world), [`test_lingbot_world_v2_stepwise.py`](../../vllm-omni/tests/e2e/online_serving/test_lingbot_world_v2_stepwise.py). InternVLA — [`test_internvla_a1.py`](../../vllm-omni/tests/examples/offline_inference/test_internvla_a1.py) (advanced level, env-gated). OpenPI transport: [`test_openpi_serving.py`](../../vllm-omni/tests/entrypoints/openai_api/test_openpi_serving.py), [`test_openpi_connection.py`](../../vllm-omni/tests/entrypoints/openai_api/test_openpi_connection.py). +- **Example commands (verbatim from READMEs; not run)** [Verified]: + - Qwen3-TTS: `python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --audio-path /path/to/reference.wav`; `vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091`; `python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you?" --stream-audio --response-format pcm --word-timestamps` ([offline README](../../vllm-omni/examples/offline_inference/text_to_speech/README.md), [online README](../../vllm-omni/examples/online_serving/text_to_speech/README.md)). + - CosyVoice3: `python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py --model pretrained_models/Fun-CosyVoice3-0.5B --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN --ref-audio /path/to/reference.wav --prompt-text "You are a helpful assistant.<|endofprompt|>Trascript in your ref audio clip"`; `vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remote-code`. + - Fish: `python examples/offline_inference/text_to_speech/fish_speech/end2end.py --text "Hello, this is a streaming test." --streaming`; `vllm serve fishaudio/s2-pro --omni --port 8091` ([recipe](../../vllm-omni/recipes/fishaudio/Fish-Speech-S2-Pro.md)). + - MOSS: `python end2end.py --model OpenMOSS-Team/MOSS-TTS --text "Hello, this is a MOSS-TTS test." --ref-audio path/to/reference.wav --output output.wav`; `vllm serve OpenMOSS-Team/MOSS-TTS-Realtime --omni --port 8091`; `vllm serve OpenMOSS-Team/MOSS-TTS-Nano --omni --port 8091` ([recipe](../../vllm-omni/recipes/OpenMOSS/MOSS-TTS.md)). + - GR00T: `vllm serve nvidia/GR00T-N1.7-3B --omni --host 127.0.0.1 --port 8000 --served-model-name gr00t-n1d7 --deploy-config vllm_omni/deploy/Gr00tN1d7.yaml`. + - π0: `vllm serve lerobot/pi0_base --omni --port 8000 --served-model-name pi0 --deploy-config vllm_omni/deploy/pi0.yaml --enforce-eager --disable-log-stats`; `python examples/online_serving/pi0/openpi_client.py --host 127.0.0.1 --port 8000 --prompt "pick up the red block and place it in the bin"` ([README](../../vllm-omni/examples/online_serving/pi0/README.md)). + - DreamZero: `vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 --served-model-name dreamzero-droid --deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml --enforce-eager --disable-log-stats`; `python examples/online_serving/dreamzero/openpi_client.py --host 127.0.0.1 --port 8000 --video-dir outputs/dreamzero/assets`. + - InternVLA-A1: `bash run.sh --num-episodes 1` in [`examples/offline_inference/internvla_a1/`](../../vllm-omni/examples/offline_inference/internvla_a1) ([README](../../vllm-omni/examples/offline_inference/internvla_a1/README.md)). + - LingBot-World: `vllm serve robbyant/lingbot-world-v2-14b-causal-fast-diffusers --omni --deploy-config vllm_omni/deploy/lingbot_world_v2_stepwise.yaml --port 8000` + [`streaming_video_client.py`](../../vllm-omni/examples/online_serving/streaming_video_generation/streaming_video_client.py). +- **Apps** [Verified]: [`apps/ComfyUI-vLLM-Omni`](../../vllm-omni/apps/ComfyUI-vLLM-Omni) (ComfyUI nodes) and [`apps/lvsa-omni-showcase`](../../vllm-omni/apps/lvsa-omni-showcase) (long-video sparse attention) — neither mobile nor TTS/VLA specific. No mobile bindings exist. + +--- + +## 7. Limitations and unknowns + +1. **No edge/CPU/mobile execution path at all** — platforms are CUDA/ROCm/NPU/XPU/MUSA only; `cpu` requirements exist solely for docs builds. [Verified] +2. **All talkers depend on vLLM's paged-attention runtime** (`Qwen2Model`/`Qwen3Model`, `ParallelLMHead`, `LogitsProcessor`); exporting them requires re-implementing embedding/head extensions on a different LLM runtime. [Verified] +3. **Stage hops serialize tensors through CPU shared memory** (msgpack), so even co-located stages pay D2H/H2D per chunk. [Verified] Latency impact not measured here. [Unknown] +4. **MOSS Delay-family streaming is inert** (`chunk_frames = 1<<30`), despite YAML `codec_streaming: true`; `MOSS_TTS_CODEC_PATH` is documented but unimplemented; Realtime online serving passes `prompt_audio_array` the talker never reads. [Verified] +5. **CosyVoice3 streaming re-decodes the full history per chunk** (cumulative CFM + HiFT), O(n²); stage 1 is not batched; `codec_left_context_frames` is unused by its processor; two RAS implementations diverge (`common/nucleus_ras_sampling.py` vs inline). [Verified] +6. **Qwen3-TTS `stop_token_ids [2150]` vs config default `codec_eos_token_id=4198`** — checkpoint override assumed. [Inferred] Actual talker layer counts for shipped checkpoints not in tree. [Unknown] +7. **Fish `dac_utils.py` docstring says the DAC encoder runs on CPU; code runs it on the stage-0 GPU.** Recipe links to non-existent doc/example paths. [Verified] +8. **VLA models have no cross-step reuse**: GR00T and π0 recompute the full VLM prefill each control step; only DreamZero/LingBot keep session KV, and those require the experimental `ARDiffusionEngine` with `max_num_seqs=1`. [Verified] +9. **GR00T warm-up never runs the model** (dummy zeros), so first-request cold start is unmitigated; `policy_server_config.image_resolution [180,320]` is advertised but the model letterboxes to 256². `final_output_type` for GR00T resolves to `"image"` in the diffusion metadata path while the pipeline declares `"actions"`. [Verified] +10. **InternVLA-A1 has no serving path** despite the supported-models table's online ✅; `end2end.py` may pass a wrong kwarg (`prompts=`). [Verified / Inferred] +11. **Parameter counts are never computed in-tree**; all sizes here are HF ids, YAML headers, or comments. DreamZero's UMT5-XXL / ViT-H sizes, GLM-TTS, IndexTTS2, OmniVoice, MiniCPM-o 4.5, PersonaPlex, Local v1.5 backbone: [Unknown]. +12. **Performance numbers are repo-reported only** (Qwen3-TTS TTFP 64 ms, MOSS Realtime "TTFB ~180 ms", InternVLA 38 s/sample); nothing reproduced. [Verified labels] +13. **DreamZero NPU/Intel checkmarks** in `supported_models.md` are not evidenced by DreamZero-specific platform code (CUDA stream sync in CFG-parallel, `torch.cuda.is_available` gates). [Inferred] +14. **Several e2e tests are skipped** (MOSS #4700/#6417); GR00T recipe references a test file that does not exist (`test_gr00t_openpi.py`). [Verified] +15. **Subagent-derived line numbers** were spot-checked but not all re-read individually; the mechanical check below covers path existence, not line accuracy. [Inferred] + +--- + +## 8. Reference index + +All links resolve relative to `analysis/engines/`; existence verified mechanically at write time. + +| Path | Role | +|---|---| +| [`vllm_omni/config/pipeline_registry.py`](../../vllm-omni/vllm_omni/config/pipeline_registry.py) | `OMNI_PIPELINES` model_type → `PipelineConfig` map, `register_pipeline`, `resolve_pipeline_config` | +| [`vllm_omni/config/stage_config.py`](../../vllm-omni/vllm_omni/config/stage_config.py) | `StageExecutionType`, `StagePipelineConfig`, `PipelineConfig`, `StageDeployConfig`, `DeployConfig`, scheduler resolution, processor-func selection | +| [`vllm_omni/model_executor/models/registry.py`](../../vllm-omni/vllm_omni/model_executor/models/registry.py) | `_OMNI_MODELS` HF-arch → module/class map | +| [`vllm_omni/model_executor/models/output_templates.py`](../../vllm-omni/vllm_omni/model_executor/models/output_templates.py) | `OmniOutput(text_hidden_states, multimodal_outputs, ...)` | +| [`vllm_omni/data_entry_keys.py`](../../vllm-omni/vllm_omni/data_entry_keys.py) | `OmniPayload` / `OmniPayloadStruct` inter-stage schema, tensor (de)serialization to CPU | +| [`vllm_omni/deploy/`](../../vllm-omni/vllm_omni/deploy) | 90 bundled deploy YAMLs (`qwen3_tts.yaml`, `cosyvoice3.yaml`, `fish_qwen3_omni*.yaml`, `moss_*.yaml`, `Gr00tN1d7.yaml`, `pi0.yaml`, `dreamzero*.yaml`, `lingbot_world_v2_stepwise.yaml`, …) | +| [`vllm_omni/model_executor/models/qwen3_tts/`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts) | talker, code2wav, prompt builder, code predictor wrapper, 12 Hz/25 Hz tokenizers, CUDA-graph wrappers | +| [`vllm_omni/model_executor/models/common/qwen3_code_predictor.py`](../../vllm-omni/vllm_omni/model_executor/models/common/qwen3_code_predictor.py) | shared residual-codebook predictor (`CodePredictorWrapper`) | +| [`vllm_omni/model_executor/stage_input_processors/qwen3_tts.py`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py) | `talker2code2wav_async_chunk`, `_full_payload`, `_token_only` | +| [`vllm_omni/utils/forced_aligner.py`](../../vllm-omni/vllm_omni/utils/forced_aligner.py) | injects Qwen3-ForcedAligner pooling stage for word timestamps | +| [`vllm_omni/model_executor/models/cosyvoice3/`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3) | talker (`CosyVoice3Model`, RAS sampler), code2wav (CFM + HiFT), TRT helpers, tokenizer, utils | +| [`vllm_omni/diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py`](../../vllm-omni/vllm_omni/diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py) | CFM estimator DiT used by CosyVoice3 code2wav | +| [`vllm_omni/model_executor/stage_input_processors/cosyvoice3.py`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/cosyvoice3.py) | CosyVoice3 async-chunk / sync processors | +| [`vllm_omni/model_executor/models/fish_speech/`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech) | slow AR, fast AR, DAC decoder/encoder, prompt utils | +| [`vllm_omni/model_executor/stage_input_processors/fish_speech.py`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/fish_speech.py) | `slow_ar_to_dac_decoder_async_chunk` | +| [`vllm_omni/attention/fish_kvcache_backend.py`](../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py) | optional Triton decode-only KV backend for Fish slow AR | +| [`vllm_omni/utils/speaker_cache.py`](../../vllm-omni/vllm_omni/utils/speaker_cache.py) | LRU speaker/voice cache shared by TTS families | +| [`vllm_omni/model_executor/models/moss_tts/`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts) | Delay/Realtime/Local talkers, depth transformer, codec decoder, audio tokenizer v1/v2, graph wrappers, reference encoder | +| [`vllm_omni/model_executor/models/moss_tts_nano/`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano) | single-stage Nano wrapper around HF remote code | +| [`vllm_omni/model_executor/stage_input_processors/moss_tts.py`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/moss_tts.py) | delay/raw async processors, sync `talker2codec` | +| [`vllm_omni/core/sched/omni_generation_scheduler.py`](../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py) | generation-stage scheduler (`retains_state_across_chunks` accounting) | +| [`vllm_omni/worker/gpu_ar_model_runner.py`](../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py) | AR runner: `talker_mtp` CUDA-graph capture, model-owned sampler hooks | +| [`vllm_omni/model_executor/models/gr00t/pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py) | `GR00T_N1D7_PIPELINE` | +| [`vllm_omni/diffusion/models/gr00t/`](../../vllm-omni/vllm_omni/diffusion/models/gr00t) | `pipeline_gr00t.py`, `policy.py`, `modeling/gr00t_n1d7.py`, `modeling/modules/dit.py`, `embodiment_conditioned_mlp.py`, `processing_gr00t_n1d7.py`, `dataio/` | +| [`vllm_omni/diffusion/models/pi0_pipeline_config.py`](../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py) | `PI0_PIPELINE` | +| [`vllm_omni/diffusion/models/pi0/`](../../vllm-omni/vllm_omni/diffusion/models/pi0) | `pipeline_pi0.py`, `modeling_pi0.py`, `processor_pi0.py`, `config.py` | +| [`vllm_omni/model_executor/models/dreamzero/pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/dreamzero/pipeline.py) | `DREAMZERO_PIPELINE` | +| [`vllm_omni/diffusion/models/dreamzero/`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero) | `pipeline_dreamzero.py`, `causal_wan_model.py`, `image_encoder.py`, `action_encoder.py`, `state_dreamzero.py`, `transform/droid.py`, `utils.py` | +| [`vllm_omni/model_executor/models/lingbot_world/pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/lingbot_world/pipeline.py) | `LINGBOT_WORLD_PIPELINE` | +| [`vllm_omni/diffusion/models/lingbot_world/`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world) | `pipeline.py`, `transformer.py`, `dmd_block.py`, `camera.py`, `actions.py` | +| [`vllm_omni/diffusion/models/internvla_a1/`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1) | `pipeline_internvla_a1.py`, `model_internvla_a1.py`, `cosmos_ci_torch.py`, `config.py` | +| [`vllm_omni/diffusion/models/cosmos3/action.py`](../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py) | Cosmos3 action-policy modes | +| [`vllm_omni/diffusion/request.py`](../../vllm-omni/vllm_omni/diffusion/request.py) | `OmniDiffusionRequest` | +| [`vllm_omni/inputs/data.py`](../../vllm-omni/vllm_omni/inputs/data.py) | `OmniDiffusionSamplingParams.extra_args` | +| [`vllm_omni/diffusion/data.py`](../../vllm-omni/vllm_omni/diffusion/data.py) | `DiffusionOutput`, `OmniDiffusionConfig`, model-class auto-detection | +| [`vllm_omni/diffusion/output_formatter.py`](../../vllm-omni/vllm_omni/diffusion/output_formatter.py) | maps `actions/audio/trajectory` into `multimodal_output` | +| [`vllm_omni/outputs/__init__.py`](../../vllm-omni/vllm_omni/outputs/__init__.py) | `OmniRequestOutput` (stage_id, final_output_type, images, latents, multimodal_output) | +| [`vllm_omni/entrypoints/openpi/connection.py`](../../vllm-omni/vllm_omni/entrypoints/openpi/connection.py) | OpenPI msgpack WebSocket connection handler | +| [`vllm_omni/entrypoints/openpi/serving.py`](../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py) | builds `OmniDiffusionRequest` with `extra_args["robot_obs"]`, extracts actions | +| [`vllm_omni/entrypoints/openai/api_server.py`](../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py) | route registration (`/v1/realtime/robot/openpi`, `/v1/realtime/video`, speech endpoints) | +| [`vllm_omni/entrypoints/openai/serving_speech.py`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py) | `/v1/audio/speech` request building per family | +| [`vllm_omni/entrypoints/openai/serving_speech_stream.py`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py) | WebSocket streaming-text TTS session, abort on disconnect | +| [`vllm_omni/entrypoints/openai/audio_utils_mixin.py`](../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py) | wav/mp3/flac/pcm/opus encoding via soundfile, torchaudio resampling | +| [`vllm_omni/entrypoints/openai/tts_adapters/`](../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters) | per-family TTS request adapters and capability loaders | +| [`vllm_omni/entrypoints/async_omni.py`](../../vllm-omni/vllm_omni/entrypoints/async_omni.py) | `AsyncOmni` generate/abort | +| [`vllm_omni/engine/async_omni_engine.py`](../../vllm-omni/vllm_omni/engine/async_omni_engine.py) | engine composition root, `abort_async`, default diffusion stage cfg | +| [`vllm_omni/engine/orchestrator.py`](../../vllm-omni/vllm_omni/engine/orchestrator.py) | cross-stage routing, `_handle_abort`, custom input func invocation | +| [`vllm_omni/engine/stage_engine_core_client.py`](../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py) | applies `custom_process_input_func` per stage | +| [`vllm_omni/distributed/omni_connectors/factory.py`](../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py) | connector registry | +| [`vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py`](../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py) | async-chunk put/get lifecycle | +| [`vllm_omni/distributed/omni_connectors/utils/serialization.py`](../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py) | `OmniSerializer` (msgpack, tensors → CPU bytes) | +| [`vllm_omni/platforms/__init__.py`](../../vllm-omni/vllm_omni/platforms/__init__.py) | platform plugin probing (cuda/rocm/npu/xpu/musa) | +| [`vllm_omni/diffusion/distributed/a2a_permute.py`](../../vllm-omni/vllm_omni/diffusion/distributed/a2a_permute.py) | only compiled CUDA source (JIT) in tree | +| [`vllm_omni/experimental/ar_diffusion/`](../../vllm-omni/vllm_omni/experimental/ar_diffusion) | `ARDiffusionEngine`, runner, session, paged KV for world models | +| [`vllm_omni/experimental/world_models/`](../../vllm-omni/vllm_omni/experimental/world_models) | session state manager and DreamZero adapter | +| [`setup.py`](../../vllm-omni/setup.py) | platform-aware requirements routing, no extensions | +| [`pyproject.toml`](../../vllm-omni/pyproject.toml) | optional extras, pytest markers | +| [`requirements/common.txt`](../../vllm-omni/requirements/common.txt) | core dependency pins | +| [`docs/design/architecture_overview.md`](../../vllm-omni/docs/design/architecture_overview.md) | stage-based architecture, Qwen3-Omni stage sizes | +| [`docs/configuration/stage_configs.md`](../../vllm-omni/docs/configuration/stage_configs.md) | pipeline vs deploy config schema | +| [`docs/design/feature/async_chunk.md`](../../vllm-omni/docs/design/feature/async_chunk.md) | async chunk design + repo-reported perf table | +| [`docs/design/feature/realtime_ar_diffusion.md`](../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md) | AR-diffusion session contract | +| [`docs/design/fullduplex.md`](../../vllm-omni/docs/design/fullduplex.md) | full-duplex runtime | +| [`docs/design/qwen3_omni_tts_performance_optimization.md`](../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md) | repo-reported Qwen3-TTS/Omni latency numbers | +| [`docs/features/session_state_manager.md`](../../vllm-omni/docs/features/session_state_manager.md) | per-session state accounting (DreamZero) | +| [`docs/serving/speech_api.md`](../../vllm-omni/docs/serving/speech_api.md) | `/v1/audio/speech`, streaming formats, voices | +| [`docs/serving/openpi_api.md`](../../vllm-omni/docs/serving/openpi_api.md) | OpenPI robot WebSocket protocol | +| [`docs/serving/streaming_video_output_api.md`](../../vllm-omni/docs/serving/streaming_video_output_api.md) | `/v1/realtime/video` | +| [`docs/models/supported_models.md`](../../vllm-omni/docs/models/supported_models.md) | model/platform support table | +| [`docs/getting_started/installation/gpu/cuda.inc.md`](../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md) | `vllm==0.28.0` pin | +| [`docs/contributing/README.md`](../../vllm-omni/docs/contributing/README.md) | test levels and marks | +| [`docs/user_guide/examples/offline_inference/internvla_a1.md`](../../vllm-omni/docs/user_guide/examples/offline_inference/internvla_a1.md) | InternVLA-A1 usage + repo-reported timings | +| [`recipes/Qwen/Qwen3-TTS.md`](../../vllm-omni/recipes/Qwen/Qwen3-TTS.md), [`recipes/fishaudio/Fish-Speech-S2-Pro.md`](../../vllm-omni/recipes/fishaudio/Fish-Speech-S2-Pro.md), [`recipes/OpenMOSS/MOSS-TTS.md`](../../vllm-omni/recipes/OpenMOSS/MOSS-TTS.md), [`recipes/OpenBMB/VoxCPM2.md`](../../vllm-omni/recipes/OpenBMB/VoxCPM2.md), [`recipes/rednote-hilab/dots.tts.md`](../../vllm-omni/recipes/rednote-hilab/dots.tts.md), [`recipes/NVIDIA/Nemotron-Labs-Audex.md`](../../vllm-omni/recipes/NVIDIA/Nemotron-Labs-Audex.md) | TTS deployment recipes (size labels, commands) | +| [`recipes/NVIDIA/GR00T-N1.7.md`](../../vllm-omni/recipes/NVIDIA/GR00T-N1.7.md), [`recipes/lerobot/Pi0.md`](../../vllm-omni/recipes/lerobot/Pi0.md), [`recipes/Robbyant/LingBot-World-2.0.md`](../../vllm-omni/recipes/Robbyant/LingBot-World-2.0.md) | VLA / world-model recipes | +| [`examples/offline_inference/text_to_speech/`](../../vllm-omni/examples/offline_inference/text_to_speech), [`examples/online_serving/text_to_speech/`](../../vllm-omni/examples/online_serving/text_to_speech) | per-family TTS examples and READMEs | +| [`examples/online_serving/pi0/`](../../vllm-omni/examples/online_serving/pi0), [`examples/online_serving/dreamzero/`](../../vllm-omni/examples/online_serving/dreamzero), [`examples/offline_inference/dreamzero/`](../../vllm-omni/examples/offline_inference/dreamzero), [`examples/offline_inference/internvla_a1/`](../../vllm-omni/examples/offline_inference/internvla_a1), [`examples/online_serving/streaming_video_generation/streaming_video_client.py`](../../vllm-omni/examples/online_serving/streaming_video_generation/streaming_video_client.py) | VLA / world-model clients | +| [`benchmarks/tts/`](../../vllm-omni/benchmarks/tts) | universal TTS benchmark CLI + model registry | +| [`tests/`](../../vllm-omni/tests) | see §6 for per-family test files | +| [`.buildkite/`](../../vllm-omni/.buildkite) | CI lanes (cuda/amd/npu/intel) | +| [`apps/`](../../vllm-omni/apps) | ComfyUI nodes, LVSA showcase (no mobile) |