[Edge] Agentic evidence for the 4-bit choice: W4A16, not W4A8

Driving a phone through the Android agent harness turns section 2c's fidelity
numbers into a task outcome. Same simulated device, one run per cell:

    build          decode        "Turn on Wi-Fi."            "Set an alarm"
    bf16           41.0 tok/s    done in 4 steps             wrong app, stuck
    W4A16 g128     80.0 tok/s    done in 4 steps             right app, no completion
    W4A8  g128     86.6 tok/s    turns it on, then off (x2)  reaches Alarm, loops

Every build navigates home -> Settings -> Network -> Wi-Fi correctly. Only
some notice the switch now reads checked=true and call task_complete. W4A8 --
the fastest build, and the one the speed-first path recommends -- ends with
the task undone, on both runs.

So "W4A16 reproduces 2/12 greedy prompts and W4A8 1/12" was not a small
difference on a proxy metric: it is finishing versus reversing. The selector
now says so where someone choosing a build will read it, and points at
priority="fidelity" for agentic work.

Two tasks, one run per cell: enough to show the failure reproduces, not to
rank success rates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
tzhouamandClaude Opus 5 committed 2026-09-12 16:31:36 +08:00
1 parent c4e61133fb
commit bc7f8237e3
1 file changed
+12
+12
View File
@@ -42,6 +42,18 @@ Fidelity is the reason both 4-bit rows stay: against each build's own bf16
output over 12 greedy prompts, llama.cpp's Q4_K_M reproduced 4/12 exactly,
w4a16 2/12 and w4a8 1/12. W4A8 pins the zero point at 8 (so the grid must be
symmetric) *and* quantizes activations to int8; W4A16 does neither.
That difference has a task-level consequence, which is why ``priority`` is not
a matter of taste. Driving a phone through the Android agent harness
(``analysis/experiments/spark_edge/android_agent``), every build navigates a
multi-screen task correctly -- and only some of them notice they have arrived.
W4A8 turned Wi-Fi on and then tapped the switch off again, on both runs;
W4A16 called ``task_complete`` citing the switch reading ``checked=true``, and
on a second task it picked the right app where bf16 picked the wrong one.
**Navigation survives quantization; recognising completion does not.** For
agentic work choose ``priority="fidelity"``; the 8% of decode it costs buys
the difference between finishing a task and undoing it. (Two tasks, one run
per cell -- enough to show the failure reproduces, not to rank success rates.)
"""
from dataclasses import dataclass, field