From bc7f8237e356089c601208ecf730f8041ffdef59 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Sat, 12 Sep 2026 16:31:36 +0800 Subject: [PATCH] [Edge] Agentic evidence for the 4-bit choice: W4A16, not W4A8 Driving a phone through the Android agent harness turns section 2c's fidelity numbers into a task outcome. Same simulated device, one run per cell: build decode "Turn on Wi-Fi." "Set an alarm" bf16 41.0 tok/s done in 4 steps wrong app, stuck W4A16 g128 80.0 tok/s done in 4 steps right app, no completion W4A8 g128 86.6 tok/s turns it on, then off (x2) reaches Alarm, loops Every build navigates home -> Settings -> Network -> Wi-Fi correctly. Only some notice the switch now reads checked=true and call task_complete. W4A8 -- the fastest build, and the one the speed-first path recommends -- ends with the task undone, on both runs. So "W4A16 reproduces 2/12 greedy prompts and W4A8 1/12" was not a small difference on a proxy metric: it is finishing versus reversing. The selector now says so where someone choosing a build will read it, and points at priority="fidelity" for agentic work. Two tasks, one run per cell: enough to show the failure reproduces, not to rank success rates. Co-Authored-By: Claude Opus 5 (1M context) --- vllm_omni/edge/weight_path.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/vllm_omni/edge/weight_path.py b/vllm_omni/edge/weight_path.py index 9b44dca63..278af084b 100644 --- a/vllm_omni/edge/weight_path.py +++ b/vllm_omni/edge/weight_path.py @@ -42,6 +42,18 @@ Fidelity is the reason both 4-bit rows stay: against each build's own bf16 output over 12 greedy prompts, llama.cpp's Q4_K_M reproduced 4/12 exactly, w4a16 2/12 and w4a8 1/12. W4A8 pins the zero point at 8 (so the grid must be symmetric) *and* quantizes activations to int8; W4A16 does neither. + +That difference has a task-level consequence, which is why ``priority`` is not +a matter of taste. Driving a phone through the Android agent harness +(``analysis/experiments/spark_edge/android_agent``), every build navigates a +multi-screen task correctly -- and only some of them notice they have arrived. +W4A8 turned Wi-Fi on and then tapped the switch off again, on both runs; +W4A16 called ``task_complete`` citing the switch reading ``checked=true``, and +on a second task it picked the right app where bf16 picked the wrong one. +**Navigation survives quantization; recognising completion does not.** For +agentic work choose ``priority="fidelity"``; the 8% of decode it costs buys +the difference between finishing a task and undoing it. (Two tasks, one run +per cell -- enough to show the failure reproduces, not to rank success rates.) """ from dataclasses import dataclass, field