vllm-omni/ROADMAP.md

Roadmap

Ordered by whether the verification method is correct, not by expected size. That ordering is deliberate: this project retracted five conclusions, and in every case the fix was to the measurement rather than the code.

Each item states the measurement it starts from, what it should gain, and how that gain would be verified. An item without a verification method is not on this list.

Done

	item	result
H1	Task oracles judged by device state, with variants	alarm oracle needed a saved alarm, not a typed 7; a Wi-Fi round trip now fails the already-satisfied task
H2	Harness semantics	arm order really rotates; best respects metric direction; provenance stamped; failures recorded
H3	The three missing measurements	ready_s, peak RssAnon, memory at a filled context — all previously unmeasurable
H4	Contention guard	aggregate stolen capacity + straggler rule, on the target cores rather than host load average
H5	Regression gates	bands from observed spread; throughput explicitly ungateable at ~30% swing
H6	Fidelity metric that ranks	top-1 agreement and KL, replacing a saturating 12-prompt greedy check
H8	Retracted numbers purged from source	two modules were still shipping withdrawn figures as documentation
I0	Blocked weight repack	489 MB of load-time peak, no cost
I1	One packed weight layout	615 MB resident, at ~1.63× prefill
I2	Measure cpu_gemm_wna16	2.2× slower than tinygemm; routing was right, now for a measured reason
I7	Memory ledger, by attribution	found a 618 MB RoPE table and an 843 MB KV cache, both inside an "unexplained 1.6 GB"

Next

I6 — import cost. import vllm_omni is ~30 s against import vllm's 5.2 s, with transformers' auto image-processing (4.6 s) and the openai client (1.3 s) identifiable inside it, for a text model that needs neither. Every stage subprocess pays it again. Expect: an upper bound of ~6 s per process; deferred imports often move cost rather than remove it. Verify: ready_s, which now exists — the old timer started after imports, so the benchmark meant to prove this could not see it.

Drop the row-major embedding copy. embed_tokens.weight_packed is 128 MB, kept solely because the tinygemm layout was believed un-addressable for a single-row lookup. That premise is now known false. Verify: RssAnon plus an embedding-output SNR assertion.

I10 first step — why is the GEMM at 60%? The decode GEMM sustains 132 GB/s against a measured 222 GB/s ceiling. Part is arithmetic (tinygemm dequantizes per activation row) and part may be that each of 113 calls streams only a few MB, too little for the prefetcher to reach steady state. Attribute before optimising. Expect: unknown — a diagnosis, whose output is whether the 2.76 ms implied by the gap is reachable at all.

I3 — asymmetric W4A8. The symmetric grid is our exporter's choice, not the kernel's limit; models/spark2_5-gptq-asym-g128 already exists. The cheapest untried fidelity experiment, and it tests the stated cause of W4A8's error. Verify: fidelity_kl.py and the scored agent suite.

I5 — attention wrapper consolidation. Four ops per layer, 1.31 ms/step, of which 0.83 ms is wrapper self time; at batch 1 the cache-update loop runs two iterations while the wrapper rebuilds context and allocates a dummy tensor for compile ordering. Expect: 0.3–0.5 ms, revised down from an earlier 0.6–0.9 ms that assumed a workload never profiled. Verify: per-op profile and unprofiled step latency — one outer call that still launches two threaded kernels removes neither thread launch.

I4 — a better quantizer. Ours reproduces 2/12 greedy prompts against its own bf16; Q4_K_M manages 4/12 — but at 4.5 bits against our 4.25, so close the budget gap first (g64, or fp16 scale+zero, both config changes) before implementing error-compensated rounding. Timebox it; a null result is worth recording.

Considered and rejected

A CPU megakernel. On GPU this fuses a forward pass into one launch to remove launch overhead and the cross-kernel prefetch bubble. On CPU the analogues are OpenMP fork/join and ATen dispatch — and llama.cpp already is one: ggml_threadpool starts threads once and they walk the whole graph, synchronising at ggml_barrier through an atomic spin, where vLLM enters a separate omp parallel region per kernel.

Measured before deciding: a parallel region costs ~1.2 µs at 24 threads, and spin-waiting does not help (OMP_WAIT_POLICY=ACTIVE moved it 1.23 → 1.18 µs), so it is barrier cost rather than thread wake-up. At a few hundred regions per decode step that is 0.3–0.4 ms, 2–3% — the same money as I5, far more cheaply. The larger prize is the streaming bubble, but capturing it means replacing the execution model inside vLLM's CPU backend, which reimplements ggml. Not next.

More kernel fusion. Four attempts, four losses: vLLM's fused CPU RMSNorm is 3.6–4.5× faster in isolation and slower in situ (inductor already fuses the native form with the residual add); fused gelu_and_mul 2.7× slower at Spark's shape; cpp.min_chunk_size −6% decode; an inline gate reduction −44%.

MiniCPM-o CPU overlay. The 8 B thinker at bf16 is ~19 GB, so it buys no edge deployability. The argument for it — that three-stage parallel init should beat the two-stage 2.1× — is also not how overlap works: the bound is ΣT/max(T).

Speculative decoding for tg128. It would inflate a fixed-prompt ignore_eos benchmark without representing anything. Measure it on the agent suite or not at all.

Thread count and warm-up batch size as memory levers. 64 MB and 2 MB.

Harness work still open

• A real device in the agent loop. The toolkit has 38 tests and has never touched a phone — this host has no adb, no emulator, no /dev/kvm. The real-device path and per-task resets exist; what is untested is whether a real Settings app matches the fixtures.
• Success rates, not single runs. The build comparison that produced "navigation survives quantization, completion does not" is one run per cell. Variants now exist so repeats are trials; they need running.