Package this fork as vllm-embedding: README, docs, roadmap, harness

Turns a branch of accumulated edge/CPU work into something someone else can
read, reproduce and continue.

README.md states what the fork is for (multimodal models on edge CPUs rather
than servers), the headline memory and speed results with their costs, and why
the numbers are worth reading -- five retracted conclusions, all measurement
errors rather than coding errors. Upstream's README is preserved at
docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the
files this fork changes.

docs/edge/:
  methodology.md  the measurement discipline, which is the most transferable
                  thing here: what each of the five retractions was, and the
                  rule it left behind. Includes the two traps specific to this
                  stack -- vLLM's AOT compile cache keying on config so two
                  arms sharing an artifact load the wrong graph, and allocated
                  not being resident until mincore(2) says so.
  results.md      every measured figure with its spread and baseline, and an
                  explicit note that the four memory levers overlap and must
                  not be added.
  cpu-4bit.md     the two 4-bit paths, the packed layout as measured rather
                  than assumed, why the path choice is a config field, and the
                  GPTQ output-head trap that invalidated a whole round of W4A8
                  numbers.
  harness.md      how to run a comparison, and the two arguments that matter
                  more than they look (--context-len, --prompt-len), each of
                  which has produced a wrong conclusion by being left at its
                  default.

ROADMAP.md orders by whether the verification method is correct rather than by
expected size, lists what is done with results, what is next with expected
gains and how each would be verified, and -- at equal length -- what was
considered and rejected, including the CPU megakernel with the measurements
that rule it out for now.

benchmarks/edge_harness/ brings the tooling into the repo so the numbers are
reproducible: the grid runner, the single-run benchmarks, the memory ledger,
the fidelity metric, the Android agent suite, the quantizers, and the tests
that pin the harness's judgement. 82 tests pass from the new location.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This commit is contained in:
tzhouamandClaude Opus 5 committed 2026-09-14 13:25:32 +08:00
1 parent 289c13df68
commit b36a3431bd
32 files changed
+5641 -86

No files matched your search

@@ -0,0 +1,92 @@
# Android agent toolkit for Spark-X2.5
Lets a text-only model operate a phone: observe the accessibility tree, call
one tool, execute it on the device, observe again.
ui_tree.py uiautomator XML -> indexed element list -> back to a tap point
device.py AdbDevice (a real phone) | SimulatedDevice (scripted, in-process)
executor.py one tool call -> one device action, errors as sentences
tool_parse.py recover a call from whatever shape the model emitted
agent.py the loop: budget, no-op detection, undo detection, fallbacks
run_episode.py CLI
## Running it
Serve the model (Spark emits GLM-4.5 tool format, so the stock parsers apply):
vllm serve <model> --port 8077 --enable-auto-tool-choice \
--tool-call-parser glm45 --reasoning-parser glm45
Then:
python run_episode.py --task "Turn on Wi-Fi." --device sim # no phone needed
python run_episode.py --task "Turn on Wi-Fi." --device adb # a real phone
`pytest test_android_agent.py` runs 21 tests with no phone and no server.
## What the loop does that a plain ReAct loop does not
Each one is a response to a measured failure of *this* model, not generic
scaffolding:
- **Retry without thinking on an empty reply.** The model can reason past its
token budget without closing `<think>`; vLLM's reasoning parser only emits
content once the block closes, so the reply arrives empty with no tool call.
Raising the cap buys a longer silence. A non-thinking retry costs ~0.6 s.
- **Report actions that changed nothing.** "the screen did not change" is
information the model cannot otherwise have.
- **Report and stop on self-undo.** Tapping a switch that is already on undoes
the task. Each tap really does change the screen, so a no-change check
cannot see it; the loop watches for the *same* action returning the screen to
its previous state. Deliberately narrow -- opening a submenu and pressing
back revisits a screen and must not trip it.
- **Recover malformed tool calls.** See below.
- **A step budget**, so a confused episode ends instead of running forever.
The loop never decides a task is finished. It reports what happened and lets
the model call `task_complete`; a harness that completes tasks on the model's
behalf measures itself.
## Measured: which build can actually finish a task
Simulated phone, one run per cell, `max_steps` 10, Xeon 8480C 24 threads.
| build | decode | "Turn on Wi-Fi." | "Set an alarm for 7:30 AM." |
|---|---|---|---|
| bf16 | 41.0 tok/s | done in 4 steps | wrong app (Settings, not Clock), toggles Airplane mode, stuck |
| **W4A16 g128** | 80.0 tok/s | done in 4 steps | right app, full navigation (Clock -> Alarm -> Add -> Save), then no tool call |
| W4A8 g128 | 86.6 tok/s | turns Wi-Fi on, then toggles it off again (2 runs, same failure) | taps "Phone" twice, reaches Alarm, then loops |
Two things fall out, and both matter more than the throughput numbers:
**Navigation survives quantization; completion does not.** Every build walks
home -> Settings -> Network -> Wi-Fi correctly. Only some of them notice that
the switch now reads `checked=true` and stop. W4A8 -- the fastest build, and
the one the speed-first selector recommends -- turns Wi-Fi on and then taps it
off again, ending with the task undone.
**The fidelity number had a task-level meaning all along.** W4A8 reproduced
1/12 greedy prompts against its own bf16 and W4A16 2/12
(analysis/spark_edge_support.md 2c). That looked like a small difference on a
proxy metric. It is the difference between finishing and not.
So for agentic use the recommendation is **W4A16**, not W4A8: 8% slower, and
it is the only 4-bit build here that completed anything. `weight_path.py`'s
`priority="fidelity"` selects it.
## Honest limits
- **No phone was involved.** This host has no `adb`, no emulator and no
`/dev/kvm`. The simulator emits real uiautomator XML, so the parser, the
index-to-coordinate mapping and the stopping rules are exercised on the path
hardware drives -- but whether a real Settings app is laid out the way these
fixtures assume is untested.
- **Two tasks, one run each.** Enough to show a failure mode reproduces, not
enough to rank builds by success rate.
- **Stock `glm45` does not always parse Spark.** On 4 of 5 screens it does. On
the Wi-Fi toggle screen the model emitted
`[tap]\n{"tool": "tap", "parameters": {"index": 1}` -- unterminated, and
`tool_calls` came back empty. `tool_parse.py` recovers that and reports every
recovery, because an agent that silently repairs its model's output hides
regressions in that model. The earlier claim that the stock parsers work
"as-is" is true most of the time, which is not the same thing.
@@ -0,0 +1,152 @@
"""Android control action space for a text-only LLM agent.
The tool surface mirrors what the mainstream ADB-based harnesses expose
(droidrun/mobilerun's Portal tools, ghost-in-the-droid's touch/see tools,
AndroidWorld's `JSONAction`), so an agent written against these schemas can be
pointed at any of them. Every action is expressed over the accessibility tree
rather than pixel coordinates, because Spark-X2.5 is a text-only model: it
never sees a screenshot, it sees the uiautomator dump.
`ADB_TEMPLATES` gives the literal `adb` invocation each tool maps to, which is
what a driver executes against a real device.
"""
TOOLS = [
{
"type": "function",
"function": {
"name": "tap",
"description": (
"Tap a UI element. Use the element's index exactly as shown in "
"the accessibility tree."
),
"parameters": {
"type": "object",
"properties": {
"index": {
"type": "integer",
"description": "Index of the element in the tree",
}
},
"required": ["index"],
},
},
},
{
"type": "function",
"function": {
"name": "type_text",
"description": (
"Type text into a focused input field. Tap the field first."
),
"parameters": {
"type": "object",
"properties": {
"index": {"type": "integer", "description": "Index of the input field"},
"text": {"type": "string", "description": "Text to enter"},
},
"required": ["index", "text"],
},
},
},
{
"type": "function",
"function": {
"name": "swipe",
"description": (
"Scroll the screen. The direction is where the content should "
"move: use 'down' to see what is further down the page."
),
"parameters": {
"type": "object",
"properties": {
"direction": {
"type": "string",
"enum": ["up", "down", "left", "right"],
}
},
"required": ["direction"],
},
},
},
{
"type": "function",
"function": {
"name": "press_key",
"description": "Press a hardware or system key.",
"parameters": {
"type": "object",
"properties": {
"key": {
"type": "string",
"enum": ["back", "home", "enter", "delete", "recent"],
}
},
"required": ["key"],
},
},
},
{
"type": "function",
"function": {
"name": "open_app",
"description": "Launch an app by its visible name.",
"parameters": {
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
},
},
},
{
"type": "function",
"function": {
"name": "task_complete",
"description": (
"Call when the user's request has been fully carried out."
),
"parameters": {
"type": "object",
"properties": {
"summary": {"type": "string", "description": "What was done"}
},
"required": ["summary"],
},
},
},
]
# How each tool becomes a real device command. `{bounds}` is filled from the
# element's centre point in the accessibility tree.
ADB_TEMPLATES = {
"tap": "adb shell input tap {x} {y}",
"type_text": "adb shell input text {text_escaped}",
"swipe": "adb shell input swipe {x1} {y1} {x2} {y2} 300",
"press_key": "adb shell input keyevent {keycode}",
"open_app": "adb shell monkey -p {package} -c android.intent.category.LAUNCHER 1",
}
KEYCODES = {
"back": "KEYCODE_BACK",
"home": "KEYCODE_HOME",
"enter": "KEYCODE_ENTER",
"delete": "KEYCODE_DEL",
"recent": "KEYCODE_APP_SWITCH",
}
SYSTEM_PROMPT = (
"You control an Android phone. At each step you are given the current "
"screen as an accessibility tree: one line per element, with its index, "
"type, text and whether it is clickable. Call exactly one tool per step to "
"make progress on the user's task. Refer to elements only by the index "
"shown.\n"
# Both rules below exist because the model failed them in measurement: it
# tapped a Wi-Fi switch that already read checked=true, turning it back
# off, and never called task_complete. A switch's state is on the screen,
# so the rule can be stated in terms of what it can see.
"A switch or checkbox shows its state as checked=true or checked=false. "
"Tapping it flips that state.\n"
"Before acting, check whether the screen already shows what the task "
"asked for. If it does, call task_complete immediately and do not tap "
"anything else -- acting again would undo it."
)
@@ -0,0 +1,271 @@
"""The observe-decide-act loop.
Three things in here are not generic agent scaffolding; they are direct
responses to how this model failed when it was measured one step at a time
(analysis/spark_edge_support.md section 5):
* **A step budget, not a bigger token cap.** On the one genuinely ambiguous
screen the model reasoned past 1200 tokens without closing `<think>`, and
vLLM's reasoning parser only emits content once the block closes -- so the
reply came back empty, with no tool call at all. Raising the cap buys a
longer silence. The loop instead retries that step with thinking disabled,
which measured 0.61 s against 8.9 s, and takes the weaker-but-present answer.
* **Detecting the no-op.** The clearest wrong action was tapping a Wi-Fi
switch that was already on -- undoing the task it had just completed. An
action that leaves the screen signature unchanged is reported back as such,
because "nothing happened" is information the model cannot otherwise have.
* **Repeat suppression.** Two identical actions on an identical screen means
stuck, and a third will not help. The loop says so in the transcript and
stops, rather than burning the budget.
"""
from __future__ import annotations
import json
import time
from dataclasses import dataclass, field
from urllib import error as urlerror
from urllib import request as urlrequest
from action_space import SYSTEM_PROMPT, TOOLS
from device import Device
from executor import ActionResult, execute
from tool_parse import extract_tool_call
from ui_tree import Element, parse_ui_dump, render_tree, screen_signature
@dataclass
class AgentConfig:
base_url: str = "http://127.0.0.1:8077/v1"
model: str = "XHToken/Spark-X2.5-1.7B"
max_steps: int = 12
think: bool = True
think_fallback: bool = True
"""On an empty reply, retry the step without thinking rather than fail."""
max_tokens: int = 1024
temperature: float = 0.0
timeout: float = 900.0
history: int = 6
retries: int = 3
"""Transient HTTP failures retried before a step is abandoned."""
retry_backoff_s: float = 2.0
"""Recent actions shown back to the model."""
@dataclass
class Step:
n: int
screen: str
tool: str | None
args: dict
result: ActionResult | None
latency_s: float
thinking: bool
note: str = ""
parsed_via: str = "tool_calls"
"""Which parser recovered the call; anything but tool_calls is worth seeing."""
@dataclass
class Episode:
task: str
steps: list[Step] = field(default_factory=list)
done: bool = False
summary: str = ""
stop_reason: str = ""
@property
def n_steps(self) -> int:
return len(self.steps)
def transcript(self) -> str:
lines = [f"task: {self.task}"]
for s in self.steps:
call = f"{s.tool}({json.dumps(s.args, ensure_ascii=False)})" if s.tool else "(no tool call)"
outcome = s.result.message if s.result else s.note
if s.parsed_via not in ("tool_calls", "none"):
outcome += f" [recovered via {s.parsed_via}]"
lines.append(f" {s.n:2d}. {call:48s} -> {outcome} [{s.latency_s:.2f}s]")
lines.append(f"stopped: {self.stop_reason}")
return "\n".join(lines)
class ChatError(RuntimeError):
pass
def chat(cfg: AgentConfig, messages: list[dict], *, think: bool) -> tuple[dict, float]:
payload = {
"model": cfg.model,
"messages": messages,
"tools": TOOLS,
"tool_choice": "auto",
"temperature": cfg.temperature,
"max_tokens": cfg.max_tokens,
}
if not think:
# Spark's chat template opens <think> by default; closing it up front
# is what makes the model answer directly.
payload["chat_template_kwargs"] = {"enable_thinking": False}
req = urlrequest.Request(
f"{cfg.base_url}/chat/completions",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
)
t0 = time.perf_counter()
last: Exception | None = None
for attempt in range(max(1, cfg.retries)):
try:
with urlrequest.urlopen(req, timeout=cfg.timeout) as r:
return json.loads(r.read()), time.perf_counter() - t0
except (urlerror.URLError, TimeoutError, json.JSONDecodeError) as exc:
# A phone task is a dozen round trips; one dropped connection
# should cost a retry, not the episode.
last = exc
if attempt + 1 < max(1, cfg.retries):
time.sleep(cfg.retry_backoff_s * (attempt + 1))
raise ChatError(
f"{cfg.base_url} failed after {cfg.retries} attempts: {last}") from last
def wait_for_server(cfg: AgentConfig, timeout_s: float = 300.0) -> bool:
"""Block until the endpoint serves a model list, or give up.
Starting an episode against a server that is still loading produces a
confusing transcript full of connection errors; this turns that into one
clear wait.
"""
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
try:
with urlrequest.urlopen(f"{cfg.base_url}/models", timeout=5) as r:
if b'"id"' in r.read():
return True
except Exception:
time.sleep(2.0)
return False
def parse_response(response: dict):
"""The one call to act on, from whichever shape the model used.
Spark does not always stay in GLM-4.5 tool format -- see tool_parse -- and
on a phone a dropped step is a stalled task, so the fallbacks matter.
"""
try:
message = response["choices"][0]["message"]
except (KeyError, IndexError):
message = {}
return extract_tool_call(message)
def first_tool_call(response: dict) -> tuple[str | None, dict]:
"""Name and arguments only, for callers that do not care how it parsed."""
parsed = parse_response(response)
return parsed.name, parsed.args
def _user_message(task: str, screen: str, log: list[str], history: int) -> str:
parts = [f"Task: {task}", "", "Current screen:", screen]
if log:
parts += ["", "Actions so far:"] + [f" {line}" for line in log[-history:]]
parts += ["", "Choose the next action. Call exactly one tool."]
return "\n".join(parts)
def observe(device: Device) -> tuple[list[Element], str, str]:
elements = parse_ui_dump(device.dump())
return elements, render_tree(elements), screen_signature(elements)
def run_episode(device: Device, task: str, cfg: AgentConfig | None = None) -> Episode:
cfg = cfg or AgentConfig()
episode = Episode(task=task)
log: list[str] = []
recent: list[tuple[str, str]] = [] # (action, screen signature before it)
signature_before_last: str | None = None
last_action: str | None = None
oscillations = 0
for n in range(1, cfg.max_steps + 1):
elements, screen, signature = observe(device)
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user",
"content": _user_message(task, screen, log, cfg.history)},
]
thinking = cfg.think
response, latency = chat(cfg, messages, think=thinking)
parsed = parse_response(response)
tool, args = parsed.name, parsed.args
if tool is None and cfg.think_fallback and thinking:
# The measured failure: reasoning never closed, so the parser
# emitted nothing. A second, non-thinking pass is worth ~0.6 s.
thinking = False
retry, retry_latency = chat(cfg, messages, think=False)
latency += retry_latency
parsed = parse_response(retry)
tool, args = parsed.name, parsed.args
if tool is None:
episode.steps.append(Step(n, screen, None, {}, None, latency, thinking,
note="model returned no tool call",
parsed_via=parsed.source))
episode.stop_reason = "no tool call, even without thinking"
return episode
if (tool_key := f"{tool}:{json.dumps(args, sort_keys=True)}") and \
recent.count((tool_key, signature)) >= 2:
episode.steps.append(Step(n, screen, tool, args, None, latency, thinking,
note="repeated a no-op action",
parsed_via=parsed.source))
episode.stop_reason = (
f"stuck: {tool} repeated on an unchanged screen"
)
return episode
recent.append((tool_key, signature))
result = execute(tool, args, elements, device)
if result.ok and not result.done:
_, _, after = observe(device)
if after == signature:
# Say so out loud: the model has no other way to learn that
# its action changed nothing.
result = ActionResult(True, result.message + " -- the screen did not change",
action=result.action)
elif tool_key == last_action and after == signature_before_last:
# Oscillation: the *same* action, undoing what it just did --
# the measured failure, tapping a switch off and on forever.
# It hides from the no-change check because each individual tap
# really does change the screen. Deliberately narrow: revisiting
# a screen by other means (open a submenu, press back) is normal
# navigation and must not trip this.
result = ActionResult(
True,
result.message + " -- that undid your previous action; the "
"setting is now back where it started",
action=result.action,
)
oscillations += 1
if oscillations >= 2:
episode.steps.append(Step(n, screen, tool, args, result, latency,
thinking, parsed_via=parsed.source))
episode.stop_reason = "stuck: repeatedly undoing its own action"
return episode
signature_before_last, last_action = signature, tool_key
episode.steps.append(Step(n, screen, tool, args, result, latency, thinking,
parsed_via=parsed.source))
log.append(f"{tool} -> {result.message}")
if result.done:
episode.done = True
episode.summary = result.summary
episode.stop_reason = "task_complete"
return episode
episode.stop_reason = f"step budget exhausted ({cfg.max_steps})"
return episode
@@ -0,0 +1,354 @@
"""Where the agent's actions actually land.
Two backends behind one interface. `AdbDevice` talks to a phone over `adb`.
`SimulatedDevice` runs a small scripted phone in-process, and exists for a
reason beyond convenience: no Android device, emulator or `/dev/kvm` is
reachable from the host this was built on, so without it the loop, the tool
executor and the stopping rules would all be untested code shipped on a
promise.
The simulator emits **real uiautomator XML**, not pre-rendered text, so the
parser, the index-to-coordinate mapping and the screen-change detection are
exercised on the same path a phone would drive. What it cannot test is
whether a real Settings app is laid out the way the fixtures assume; that is
stated in the report rather than hidden.
"""
from __future__ import annotations
import shutil
import subprocess
from xml.sax.saxutils import escape as _xml_escape
import time
from dataclasses import dataclass, field
from typing import Callable, Protocol
KEYCODES = {
"back": "KEYCODE_BACK",
"home": "KEYCODE_HOME",
"enter": "KEYCODE_ENTER",
"delete": "KEYCODE_DEL",
"recent": "KEYCODE_APP_SWITCH",
}
class Device(Protocol):
def dump(self) -> str: ...
def tap(self, x: int, y: int) -> None: ...
def input_text(self, text: str) -> None: ...
def swipe(self, x1: int, y1: int, x2: int, y2: int, ms: int = 300) -> None: ...
def keyevent(self, key: str) -> None: ...
def launch(self, app: str) -> None: ...
def size(self) -> tuple[int, int]: ...
class AdbError(RuntimeError):
pass
class AdbDevice:
"""A real phone over `adb`."""
def __init__(self, serial: str | None = None, adb: str = "adb",
timeout: float = 30.0):
if shutil.which(adb) is None:
raise AdbError(
f"{adb!r} is not on PATH. Install platform-tools and connect a "
"device (`adb devices` should list it)."
)
self.adb, self.serial, self.timeout = adb, serial, timeout
self._size: tuple[int, int] | None = None
def _run(self, *args: str, binary: bool = False):
cmd = [self.adb] + (["-s", self.serial] if self.serial else []) + list(args)
proc = subprocess.run(cmd, capture_output=True, timeout=self.timeout)
if proc.returncode != 0:
raise AdbError(f"{' '.join(cmd)} failed: {proc.stderr.decode(errors='replace')[:400]}")
return proc.stdout if binary else proc.stdout.decode(errors="replace")
def dump(self) -> str:
# exec-out keeps the XML on stdout instead of round-tripping through
# /sdcard, which is both faster and avoids a writable-storage
# assumption that does not hold on every device.
out = self._run("exec-out", "uiautomator", "dump", "/dev/tty")
start = out.find("<?xml")
if start < 0:
start = out.find("<hierarchy")
if start < 0:
raise AdbError(f"no UI hierarchy in dump output: {out[:200]!r}")
end = out.rfind("</hierarchy>")
return out[start:end + len("</hierarchy>")] if end > 0 else out[start:]
def tap(self, x: int, y: int) -> None:
self._run("shell", "input", "tap", str(x), str(y))
def input_text(self, text: str) -> None:
# `input text` takes a single token: spaces become %s and the shell
# metacharacters have to go, or half the string is eaten.
escaped = text.replace(" ", "%s")
for ch in "()<>|;&*\\~\"'`$":
escaped = escaped.replace(ch, "\\" + ch)
self._run("shell", "input", "text", escaped)
def swipe(self, x1: int, y1: int, x2: int, y2: int, ms: int = 300) -> None:
self._run("shell", "input", "swipe", str(x1), str(y1), str(x2), str(y2), str(ms))
def keyevent(self, key: str) -> None:
code = KEYCODES.get(key, key)
self._run("shell", "input", "keyevent", code)
def launch(self, app: str) -> None:
"""Launch by package name, or by visible label via the launcher."""
if "." in app:
self._run("shell", "monkey", "-p", app, "-c",
"android.intent.category.LAUNCHER", "1")
return
raise AdbError(
f"open_app needs a package name, got {app!r}. Resolve the label to a "
"package first (`adb shell pm list packages`) or tap its icon."
)
def size(self) -> tuple[int, int]:
if self._size is None:
out = self._run("shell", "wm", "size")
part = out.strip().rsplit(":", 1)[-1].strip()
w, h = part.split("x")
self._size = (int(w), int(h))
return self._size
# --------------------------------------------------------------------------
# Simulated phone
# --------------------------------------------------------------------------
@dataclass
class SimElement:
label: str
cls: str = "android.widget.TextView"
clickable: bool = True
checkable: bool = False
checked: bool = False
selected: bool = False
editable: bool = False
goto: str | None = None
"""Screen this element navigates to when tapped."""
toggles: str | None = None
"""State key this element flips when tapped."""
saves: str | None = None
"""Record key. Tapping commits the current field values into
``records[saves]`` and clears them -- the difference between typing a time
and having an alarm, which the scored suite previously could not tell."""
label_of: str | None = None
"""State key whose value picks the label from ``labels``."""
labels: tuple[str, str] = ("", "")
"""(label when the state key is false, label when true)."""
def rendered_label(self, state: dict) -> str:
if self.label_of:
return self.labels[bool(state.get(self.label_of))]
return self.label
@dataclass
class SimScreen:
screen_id: str
elements: list[SimElement]
back: str | None = None
dynamic: "Callable[[SimulatedDevice], list[SimElement]] | None" = None
"""Rows derived from device state, appended after ``elements``. A saved
alarm has to appear *on the screen*, or an oracle that reads the screen --
which is the only kind a real phone can have -- cannot tell a committed
alarm from a typed one."""
@dataclass
class SimulatedDevice:
"""A scripted phone that speaks uiautomator XML.
Deliberately small. It is here to prove the loop terminates, recovers and
stops -- not to model Android.
"""
screens: dict[str, SimScreen]
current: str
state: dict[str, bool] = field(default_factory=dict)
width: int = 1080
height: int = 2400
typed: list[str] = field(default_factory=list)
actions: list[tuple[str, ...]] = field(default_factory=list)
home: str = "home"
focus: str | None = None
"""Label of the focused text field; typing lands here."""
fields: dict[str, str] = field(default_factory=dict)
records: dict[str, list[dict]] = field(default_factory=dict)
toggled: list[tuple[str, bool, bool]] = field(default_factory=list)
"""(key, before, after) per switch tap. A task whose starting state is
already correct is failed by a round trip, which final state cannot see."""
_ROW_H = 160
_TOP = 200
def _visible(self) -> list[SimElement]:
screen = self.screens[self.current]
rows = list(screen.elements)
if screen.dynamic:
rows = [r for r in rows if not getattr(r, "_placeholder", False)]
rows.extend(screen.dynamic(self))
return rows
def _bounds(self, row: int) -> tuple[int, int, int, int]:
y1 = self._TOP + row * self._ROW_H
return 0, y1, self.width, y1 + self._ROW_H
def dump(self) -> str:
rows = []
for i, el in enumerate(self._visible()):
x1, y1, x2, y2 = self._bounds(i)
checked = el.checked
if el.toggles:
checked = self.state.get(el.toggles, el.checked)
# uiautomator escapes attribute values, and labels really do
# contain them -- "Network & internet" is on every Android
# settings screen. Emitting raw breaks the parser on the second
# screen of the first task.
label = _xml_escape(el.rendered_label(self.state), {'"': "&quot;"})
rows.append(
f' <node index="{i}" text="{label}" resource-id="" '
f'class="{el.cls}" package="com.example.sim" content-desc="" '
f'checkable="{str(el.checkable).lower()}" '
f'checked="{str(checked).lower()}" '
f'clickable="{str(el.clickable).lower()}" enabled="true" '
f'focusable="{str(el.clickable).lower()}" focused="false" '
f'scrollable="false" long-clickable="false" password="false" '
f'selected="{str(el.selected).lower()}" '
f'bounds="[{x1},{y1}][{x2},{y2}]" />'
)
body = "\n".join(rows)
return (
'<?xml version="1.0" encoding="UTF-8"?>\n'
f'<hierarchy rotation="0">\n{body}\n</hierarchy>'
)
def _row_at(self, y: int) -> int | None:
row = (y - self._TOP) // self._ROW_H
return row if 0 <= row < len(self._visible()) else None
def tap(self, x: int, y: int) -> None:
self.actions.append(("tap", str(x), str(y)))
row = self._row_at(y)
if row is None:
return
el = self._visible()[row]
if el.editable:
self.focus = el.label
if el.toggles:
before = bool(self.state.get(el.toggles, el.checked))
self.state[el.toggles] = not before
self.toggled.append((el.toggles, before, not before))
if el.saves:
self.records.setdefault(el.saves, []).append(
{**self.fields, **{k: v for k, v in self.state.items()
if k.startswith(el.saves + "_")}})
self.fields = {}
self.focus = None
if el.goto and el.goto in self.screens:
self.current = el.goto
def input_text(self, text: str) -> None:
self.actions.append(("text", text))
self.typed.append(text)
if self.focus is not None:
self.fields[self.focus] = self.fields.get(self.focus, "") + text
def swipe(self, x1: int, y1: int, x2: int, y2: int, ms: int = 300) -> None:
self.actions.append(("swipe", str(x1), str(y1), str(x2), str(y2)))
def keyevent(self, key: str) -> None:
self.actions.append(("key", key))
if key == "back":
self.current = self.screens[self.current].back or self.home
elif key == "home":
self.current = self.home
def launch(self, app: str) -> None:
self.actions.append(("launch", app))
for screen in self.screens.values():
for el in screen.elements:
if el.label.lower() == app.lower() and el.goto:
self.current = el.goto
return
def size(self) -> tuple[int, int]:
return self.width, self.height
def _saved_alarm_rows(device: "SimulatedDevice") -> list[SimElement]:
"""Render committed alarms the way a clock app lists them."""
saved = device.records.get("alarm", [])
if not saved:
return [SimElement("No alarms", clickable=False)]
rows = []
for rec in saved:
hour = (rec.get("Hour") or "?").lstrip("0") or "0"
minute = (rec.get("Minute") or "?").rjust(2, "0")
meridiem = "PM" if rec.get("alarm_pm") else "AM"
rows.append(SimElement(f"{hour}:{minute} {meridiem}", clickable=False))
return rows
def demo_phone() -> SimulatedDevice:
"""A two-app phone: Settings (Wi-Fi) and Clock (alarms).
Enough to exercise a multi-screen task, a stateful toggle -- the case the
earlier single-step run got wrong, tapping a switch that was already on --
and a text field.
"""
screens = {
"home": SimScreen("home", [
SimElement("Phone"),
SimElement("Messages"),
SimElement("Clock", goto="clock"),
SimElement("Settings", goto="settings"),
SimElement("Camera"),
]),
"settings": SimScreen("settings", [
SimElement("Settings", clickable=False),
SimElement("Network & internet", goto="network"),
SimElement("Connected devices"),
SimElement("Apps"),
SimElement("Battery"),
], back="home"),
"network": SimScreen("network", [
SimElement("Network & internet", clickable=False),
SimElement("Wi-Fi", cls="android.widget.Switch", checkable=True,
toggles="wifi"),
SimElement("Mobile network"),
SimElement("Airplane mode", cls="android.widget.Switch",
checkable=True, toggles="airplane"),
], back="settings"),
"clock": SimScreen("clock", [
SimElement("Alarm", cls="android.widget.Button", goto="alarms"),
SimElement("Clock", cls="android.widget.Button", selected=True),
SimElement("Timer", cls="android.widget.Button"),
SimElement("Stopwatch", cls="android.widget.Button"),
], back="home"),
"alarms": SimScreen("alarms", [
SimElement("Alarm", cls="android.widget.Button", selected=True),
SimElement("Add alarm", cls="android.widget.ImageButton",
goto="new_alarm"),
], back="clock", dynamic=_saved_alarm_rows),
"new_alarm": SimScreen("new_alarm", [
SimElement("Set time", clickable=False),
SimElement("Hour", cls="android.widget.EditText", editable=True),
SimElement("Minute", cls="android.widget.EditText", editable=True),
# Android shows one AM/PM control whose label is its value, so
# "set an alarm for 7:30 AM" has a wrong answer as well as a right
# one. Without it the task could be passed by typing any digits.
SimElement("AM", cls="android.widget.Button",
label_of="alarm_pm", labels=("AM", "PM"),
toggles="alarm_pm"),
SimElement("Save", cls="android.widget.Button", goto="alarms",
saves="alarm"),
], back="alarms"),
}
return SimulatedDevice(screens=screens, current="home", state={"wifi": False})
@@ -0,0 +1,147 @@
"""Turn one tool call into one device action.
Every failure here is returned to the model as a sentence it can act on, not
raised. A 1.7B model asked for `index: 9` on a four-element screen has made a
recoverable mistake, and "index 9 is out of range, the screen has 4 elements
(0-3)" is the difference between the next step being right and the episode
ending. Exceptions are reserved for the device being gone.
"""
from __future__ import annotations
from dataclasses import dataclass
from device import KEYCODES, Device
from ui_tree import Element
@dataclass
class ActionResult:
ok: bool
message: str
done: bool = False
summary: str = ""
action: str = ""
def as_tool_reply(self) -> str:
return self.message
def _need(args: dict, key: str, kind: type, what: str) -> tuple[object, str | None]:
if key not in args:
return None, f"missing required argument {key!r} for {what}"
value = args[key]
if kind is int:
try:
value = int(value)
except (TypeError, ValueError):
return None, f"{key!r} must be a whole number, got {args[key]!r}"
elif not isinstance(value, kind):
return None, f"{key!r} must be {kind.__name__}, got {type(value).__name__}"
return value, None
def _resolve(index: object, elements: list[Element]) -> tuple[Element | None, str | None]:
i = int(index)
if not elements:
return None, "the screen has no addressable elements; try press_key(back)"
if not 0 <= i < len(elements):
return None, (
f"index {i} is out of range; the screen has {len(elements)} "
f"elements (0-{len(elements) - 1})"
)
return elements[i], None
def execute(name: str, args: dict, elements: list[Element],
device: Device) -> ActionResult:
"""Run one tool call. Returns what to tell the model."""
args = args or {}
if name == "task_complete":
summary = str(args.get("summary", "")).strip()
return ActionResult(True, "task marked complete", done=True,
summary=summary, action="task_complete")
if name == "tap":
index, err = _need(args, "index", int, "tap")
if err:
return ActionResult(False, err, action="tap")
el, err = _resolve(index, elements)
if err:
return ActionResult(False, err, action="tap")
x, y = el.center
device.tap(x, y)
state = ""
if "checked" in el.flags:
state = f" (was {'on' if el.flags['checked'] else 'off'})"
return ActionResult(True, f'tapped [{index}] "{el.text}"{state}',
action=f"tap:{index}")
if name == "type_text":
index, err = _need(args, "index", int, "type_text")
if err:
return ActionResult(False, err, action="type_text")
text, err = _need(args, "text", str, "type_text")
if err:
return ActionResult(False, err, action="type_text")
el, err = _resolve(index, elements)
if err:
return ActionResult(False, err, action="type_text")
# Focus first: `input text` goes to whatever holds focus, which on a
# freshly dumped screen is usually nothing.
x, y = el.center
device.tap(x, y)
device.input_text(str(text))
return ActionResult(True, f'typed "{text}" into [{index}] "{el.text}"',
action=f"type:{index}")
if name == "swipe":
direction = str(args.get("direction", "")).lower()
if direction not in ("up", "down", "left", "right"):
return ActionResult(
False, "direction must be one of up, down, left, right",
action="swipe")
w, h = device.size()
cx, cy = w // 2, h // 2
dx, dy = int(w * 0.35), int(h * 0.30)
# `direction` is where the *content* should move, i.e. what the user
# means by "scroll down to see more", so the finger travels the other
# way. Stated in the tool description too, because guessing this wrong
# is invisible in the transcript.
moves = {
"down": (cx, cy + dy, cx, cy - dy),
"up": (cx, cy - dy, cx, cy + dy),
"right": (cx + dx, cy, cx - dx, cy),
"left": (cx - dx, cy, cx + dx, cy),
}
device.swipe(*moves[direction])
return ActionResult(True, f"swiped to scroll {direction}",
action=f"swipe:{direction}")
if name == "press_key":
key = str(args.get("key", "")).lower()
if key not in KEYCODES:
return ActionResult(
False, f"unknown key {key!r}; use one of {', '.join(KEYCODES)}",
action="press_key")
device.keyevent(key)
return ActionResult(True, f"pressed {key}", action=f"key:{key}")
if name == "open_app":
app = str(args.get("name", "")).strip()
if not app:
return ActionResult(False, "open_app needs a name", action="open_app")
try:
device.launch(app)
except Exception as exc: # device-specific launch failures are recoverable
return ActionResult(False, f"could not launch {app!r}: {exc}",
action="open_app")
return ActionResult(True, f"launched {app}", action=f"open:{app}")
return ActionResult(
False,
f"unknown tool {name!r}; available: tap, type_text, swipe, press_key, "
"open_app, task_complete",
action="unknown",
)
@@ -0,0 +1,135 @@
"""What the phone is actually in, read the same way for both backends.
The scored suite used to take its predicates over ``SimulatedDevice``'s Python
attributes, so every scored run was a simulator run by construction -- the
real-device path existed for *driving* a phone and not for judging one. That
made "the agent suite" a claim about fixtures.
Success is therefore read two ways, and the order matters:
1. **Out of band, from the setting itself** -- ``device.state`` in the
simulator, ``settings get global wifi_on`` over adb. This is the ground
truth, and it does not depend on where the agent happened to leave the
phone. Scoring from the final screen alone fails an agent that turns Wi-Fi
on correctly and then presses Home, which is not a failure.
2. **From the UI dump**, for anything with no setting behind it -- a saved
alarm appears as a row in the clock app and nowhere else. The simulator
emits real uiautomator XML, so this goes down the same parse, the same
switch lookup and the same text matching a phone would.
Only the facts a phone genuinely cannot report after the fact -- which
switches were flipped on the way, and what was typed into which field -- come
from the simulator object, and any task needing them declares it, so a
real-device run refuses to score rather than scoring wrong.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from ui_tree import parse_ui_dump
def _norm(s: str) -> str:
"""Match labels across the punctuation Android is inconsistent about."""
return "".join(c for c in s.lower() if c.isalnum())
@dataclass
class PhoneState:
source: str
"""``"sim"`` or ``"device"`` -- which facts below are trustworthy."""
settings: dict[str, bool] = field(default_factory=dict)
"""Authoritative setting values, read out of band. Preferred over
:attr:`switches`, which only shows what is currently on screen."""
screen: str | None = None
"""Simulator screen id. ``None`` on a real phone, where there is no such
thing; use :meth:`has_text` against something the screen actually shows."""
switches: dict[str, bool] = field(default_factory=dict)
texts: list[str] = field(default_factory=list)
typed: list[str] = field(default_factory=list)
fields: dict[str, str] = field(default_factory=dict)
records: dict[str, list[dict]] = field(default_factory=dict)
toggled: list[tuple[str, bool, bool]] = field(default_factory=list)
"""(key, before, after) for every switch the agent actually flipped."""
def switch(self, *names: str) -> bool | None:
"""State of the first control matching any of ``names``.
The setting wins over the screen: an agent that turned Wi-Fi on and
then navigated away has still turned Wi-Fi on. ``None`` means neither
source knows the control, which is not the same as off -- an agent that
never found the screen has not turned anything on, and predicates must
not read that as "off".
"""
wanted = {_norm(n) for n in names}
for key, value in self.settings.items():
if _norm(key) in wanted:
return bool(value)
for label, checked in self.switches.items():
if _norm(label) in wanted:
return checked
return None
def has_text(self, *needles: str) -> bool:
joined = " ".join(self.texts).lower()
return any(n.lower() in joined for n in needles)
def times_toggled(self, key: str) -> int:
return sum(1 for k, _, _ in self.toggled if k == key)
def read_state(device) -> PhoneState:
"""Read the phone through its screen, plus the simulator's own records."""
elements = parse_ui_dump(device.dump())
# `checkable` is deliberately not among the flags ui_tree parses, because
# that list is also what the model is shown; adding to it would change the
# agent's prompt. A switch is recognised by its class or by exposing a
# `checked` flag instead.
switches = {e.text: bool(e.flags.get("checked", False))
for e in elements
if "checked" in e.flags
or e.short_class in ("Switch", "CheckBox", "ToggleButton")}
texts = [e.text for e in elements if e.text]
if not hasattr(device, "screens"):
return PhoneState(source="device", switches=switches, texts=texts,
settings=adb_settings(device))
return PhoneState(
source="sim",
screen=device.current,
settings={k: bool(v) for k, v in device.state.items()},
switches=switches,
texts=texts,
typed=list(device.typed),
fields=dict(getattr(device, "fields", {})),
records={k: [dict(r) for r in v]
for k, v in getattr(device, "records", {}).items()},
toggled=list(getattr(device, "toggled", [])),
)
ADB_SETTINGS = {
"wifi": ("global", "wifi_on"),
"airplane": ("global", "airplane_mode_on"),
"bluetooth": ("global", "bluetooth_on"),
}
"""Settings readable out of band, so scoring does not depend on which screen
the agent left the phone on."""
def adb_settings(device) -> dict[str, bool]:
"""Read the authoritative values from a real phone. Missing keys are
omitted rather than defaulted -- an unreadable setting must not be scored
as "off"."""
out: dict[str, bool] = {}
for key, (namespace, name) in ADB_SETTINGS.items():
try:
raw = device._run("shell", "settings", "get", namespace, name)
except Exception:
continue
raw = raw.strip()
if raw in ("0", "1"):
out[key] = raw == "1"
return out
@@ -0,0 +1,98 @@
"""Drive Spark-X2.5 through Android control steps against an OpenAI endpoint.
Measures whether the model picks the right tool and the right element, and how
long a decision takes -- the two things that decide whether a 1.7B model can
actually run a phone agent locally.
"""
import argparse, json, time
from urllib import request as urlrequest
from action_space import SYSTEM_PROMPT, TOOLS
from scenarios import SCENARIOS
def chat(base_url, model, messages, tools, timeout=900, max_tokens=1024,
think=True):
payload = {
"model": model, "messages": messages, "tools": tools,
"tool_choice": "auto", "temperature": 0.0, "max_tokens": max_tokens,
}
if not think:
# Spark's template opens <think> by default; closing it up front makes
# the model answer directly, which is what a phone agent wants.
payload["chat_template_kwargs"] = {"enable_thinking": False}
body = json.dumps(payload).encode()
req = urlrequest.Request(f"{base_url}/chat/completions", data=body,
headers={"Content-Type": "application/json"})
t0 = time.perf_counter()
with urlrequest.urlopen(req, timeout=timeout) as r:
out = json.loads(r.read())
return out, time.perf_counter() - t0
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--base-url", default="http://127.0.0.1:8077/v1")
ap.add_argument("--model", default="XHToken/Spark-X2.5-1.7B")
ap.add_argument("--no-think", action="store_true")
ap.add_argument("--max-tokens", type=int, default=1024)
ap.add_argument("--out", default="android_agent_results.json")
a = ap.parse_args()
rows, correct_tool, correct_args = [], 0, 0
for sc in SCENARIOS:
msgs = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content":
f"Task: {sc['task']}\n\nCurrent screen:\n{sc['screen']}\n\n"
"Choose the next action."},
]
try:
out, dt = chat(a.base_url, a.model, msgs, TOOLS,
max_tokens=a.max_tokens, think=not a.no_think)
except Exception as e:
rows.append({"id": sc["id"], "error": str(e)[:200]})
print(f"{sc['id']:<20} ERROR {str(e)[:90]}", flush=True)
continue
msg = out["choices"][0]["message"]
calls = msg.get("tool_calls") or []
got_name = calls[0]["function"]["name"] if calls else None
try:
got_args = json.loads(calls[0]["function"]["arguments"]) if calls else {}
except Exception:
got_args = {"_raw": calls[0]["function"]["arguments"]}
name_ok = got_name in sc["accept_names"]
args_ok = name_ok and all(
str(got_args.get(k)) == str(v) for k, v in sc["expect"]["args"].items()
) if got_name == sc["expect"]["name"] else name_ok and not sc["expect"]["args"]
correct_tool += int(name_ok)
correct_args += int(bool(args_ok))
rows.append({
"id": sc["id"], "latency_s": round(dt, 2),
"expected": sc["expect"], "got_name": got_name, "got_args": got_args,
"tool_ok": name_ok, "args_ok": bool(args_ok),
"reasoning_chars": len(msg.get("reasoning_content") or ""),
"completion_tokens": out.get("usage", {}).get("completion_tokens"),
})
flag = "OK " if args_ok else ("~ " if name_ok else "X ")
print(f"{flag}{sc['id']:<20} {got_name}({got_args}) {dt:.1f}s", flush=True)
n = len(SCENARIOS)
summary = {
"scenarios": n,
"thinking": not a.no_think,
"tool_accuracy": f"{correct_tool}/{n}",
"action_accuracy": f"{correct_args}/{n}",
"median_latency_s": round(
sorted(r["latency_s"] for r in rows if "latency_s" in r)[
max(0, len([r for r in rows if "latency_s" in r]) // 2)] , 2)
if any("latency_s" in r for r in rows) else None,
"rows": rows,
}
json.dump(summary, open(a.out, "w"), indent=2)
print(json.dumps({k: v for k, v in summary.items() if k != "rows"}, indent=2))
if __name__ == "__main__":
main()
@@ -0,0 +1,106 @@
"""Drive Spark-X2.5 through a whole phone task, not one decision.
# against the in-process simulated phone (no device needed)
python run_episode.py --task "Turn on Wi-Fi." --device sim
# against a real phone
adb devices # confirm it is listed
python run_episode.py --task "Turn on Wi-Fi." --device adb
The server side is vLLM's stock OpenAI endpoint; Spark emits GLM-4.5's tool
format, so no custom parser is needed:
vllm serve <model> --port 8077 --enable-auto-tool-choice \\
--tool-call-parser glm45 --reasoning-parser glm45
"""
from __future__ import annotations
import argparse
import json
import sys
import time
from agent import AgentConfig, run_episode
from device import AdbDevice, AdbError, demo_phone
TASKS = {
"wifi": "Turn on Wi-Fi.",
"alarm": "Set an alarm for 7:30 AM.",
}
def build_device(kind: str, serial: str | None):
if kind == "sim":
return demo_phone(), "simulated phone (settings + clock)"
try:
device = AdbDevice(serial=serial)
w, h = device.size()
return device, f"adb device {serial or '(default)'} {w}x{h}"
except AdbError as exc:
print(f"error: {exc}", file=sys.stderr)
raise SystemExit(2)
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--task", default=TASKS["wifi"],
help=f"free text, or one of: {', '.join(TASKS)}")
ap.add_argument("--device", choices=("sim", "adb"), default="sim")
ap.add_argument("--serial", default=None)
ap.add_argument("--base-url", default="http://127.0.0.1:8077/v1")
ap.add_argument("--model", default="XHToken/Spark-X2.5-1.7B")
ap.add_argument("--max-steps", type=int, default=12)
ap.add_argument("--max-tokens", type=int, default=1024)
ap.add_argument("--no-think", action="store_true")
ap.add_argument("--no-think-fallback", action="store_true")
ap.add_argument("--out", default=None)
a = ap.parse_args()
task = TASKS.get(a.task, a.task)
device, described = build_device(a.device, a.serial)
cfg = AgentConfig(
base_url=a.base_url, model=a.model, max_steps=a.max_steps,
max_tokens=a.max_tokens, think=not a.no_think,
think_fallback=not a.no_think_fallback,
)
print(f"device: {described}")
print(f"task: {task}")
t0 = time.perf_counter()
episode = run_episode(device, task, cfg)
wall = time.perf_counter() - t0
print()
print(episode.transcript())
latencies = [s.latency_s for s in episode.steps]
print(f"\n{episode.n_steps} steps in {wall:.1f} s"
f" (median decision {sorted(latencies)[len(latencies) // 2]:.2f} s)"
if latencies else f"\n{episode.n_steps} steps in {wall:.1f} s")
if hasattr(device, "state"):
print(f"device state: {device.state}")
if a.out:
payload = {
"task": task, "device": described, "done": episode.done,
"summary": episode.summary, "stop_reason": episode.stop_reason,
"wall_s": round(wall, 2),
"think": cfg.think, "max_steps": cfg.max_steps,
"steps": [
{"n": s.n, "tool": s.tool, "args": s.args,
"ok": s.result.ok if s.result else False,
"message": s.result.message if s.result else s.note,
"latency_s": round(s.latency_s, 3), "thinking": s.thinking}
for s in episode.steps
],
"final_state": getattr(device, "state", None),
}
with open(a.out, "w") as f:
json.dump(payload, f, indent=2, ensure_ascii=False)
print(f"wrote {a.out}")
raise SystemExit(0 if episode.done else 1)
if __name__ == "__main__":
main()
@@ -0,0 +1,151 @@
"""Run the task suite and score it, so an agent change can be compared.
Success is decided by the phone's own settings and screen, not by the
transcript: an agent that says it turned Wi-Fi on and left it off fails here,
which is exactly the failure this model exhibits.
Two things changed after the plan review:
* ``--repeat N`` now walks the task's **variants** rather than re-running one
fixture. At temperature 0 against fixed fixtures, repeats measure
repeatability; they only become trials if the phone differs between them.
The summary reports a pass rate per task with the variants named, so a build
comparison rests on more than one cell.
* ``--device adb`` scores a real phone. Each run resets the device to the
task's declared starting state first, and a task that cannot be scored from
a phone (because its predicate needs flip history) is skipped and reported
as skipped, rather than silently scored from final state.
"""
from __future__ import annotations
import argparse
import json
import statistics
import time
from collections import defaultdict
from agent import AgentConfig, run_episode, wait_for_server
from phone_state import read_state
from tasks import SUITE, SUITE_BY_ID
def _reset_real_device(device, task) -> None:
if not task.reset_adb:
raise SystemExit(
f"task {task.task_id!r} declares no reset_adb, so a phone run would "
f"be scored against an unknown starting state. Add one or exclude "
f"the task with --tasks."
)
for cmd in task.reset_adb:
device._run(*cmd)
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--base-url", default="http://127.0.0.1:8077/v1")
ap.add_argument("--model", default="spark")
ap.add_argument("--tasks", default="all",
help="comma-separated task ids, or 'all'")
ap.add_argument("--repeat", type=int, default=1,
help="runs per task; each walks the next variant")
ap.add_argument("--device", choices=("sim", "adb"), default="sim")
ap.add_argument("--serial", default=None, help="adb device serial")
ap.add_argument("--no-think", action="store_true")
ap.add_argument("--max-tokens", type=int, default=1024)
ap.add_argument("--label", default="")
ap.add_argument("--out", default=None)
a = ap.parse_args()
tasks = SUITE if a.tasks == "all" else [SUITE_BY_ID[t] for t in a.tasks.split(",")]
cfg = AgentConfig(base_url=a.base_url, model=a.model,
think=not a.no_think, max_tokens=a.max_tokens)
phone = None
if a.device == "adb":
from device import AdbDevice
phone = AdbDevice(serial=a.serial)
skipped = [t.task_id for t in tasks if t.needs]
if skipped:
print(f"skipping on a real phone (needs flip history): "
f"{', '.join(skipped)}")
tasks = [t for t in tasks if not t.needs]
if not wait_for_server(cfg, timeout_s=600):
raise SystemExit(f"no model server at {a.base_url}")
rows, t0 = [], time.perf_counter()
for task in tasks:
for run in range(a.repeat):
if phone is not None:
_reset_real_device(phone, task)
device = phone
else:
device = task.build(run)
cfg.max_steps = task.max_steps
episode = run_episode(device, task.prompt, cfg)
score = task.score(read_state(device), episode.done, run)
score.update(
run=run + 1, steps=episode.n_steps,
stop_reason=episode.stop_reason,
recovered_steps=sum(1 for s in episode.steps
if s.parsed_via not in ("tool_calls", "none")),
median_latency_s=round(
sorted(s.latency_s for s in episode.steps)[episode.n_steps // 2], 2)
if episode.n_steps else None,
)
rows.append(score)
mark = "PASS" if score["success"] else "fail"
detail = []
if not score.get("scorable", True):
mark, detail = "skip", [score.get("reason", "")]
else:
if score["reached_goal"] and not score["declared_done"]:
detail.append("goal reached but never stopped")
if score["declared_done"] and not score["reached_goal"]:
detail.append("claimed done, goal NOT reached")
if score["collateral_damage"]:
detail.append("collateral damage")
print(f" {mark} {task.task_id:18s} {score['variant']:10s} "
f"steps={score['steps']:2d} "
f"{score['stop_reason'][:34]:36s} {'; '.join(detail)}", flush=True)
wall = time.perf_counter() - t0
scored = [r for r in rows if r.get("scorable", True)]
n_pass = sum(1 for r in scored if r["success"])
reached = sum(1 for r in scored if r["reached_goal"])
claimed = sum(1 for r in scored if r["declared_done"] and not r["reached_goal"])
by_task: dict[str, list[dict]] = defaultdict(list)
for r in scored:
by_task[r["task_id"]].append(r)
print("\npass rate per task")
for task_id, rs in by_task.items():
passes = sum(1 for r in rs if r["success"])
failed = [r["variant"] for r in rs if not r["success"]]
print(f" {task_id:18s} {passes}/{len(rs)}"
+ (f" failed on: {', '.join(failed)}" if failed else ""))
rates = [sum(1 for r in rs if r["success"]) / len(rs) for rs in by_task.values()]
spread = (f"{min(rates):.0%}-{max(rates):.0%}" if rates else "-")
print(f"\n{n_pass}/{len(scored)} passed in {wall:.0f}s"
f" (goal reached {reached}/{len(scored)}; "
f"falsely claimed done {claimed}; per-task rate {spread}"
+ (f"; median {statistics.median(rates):.0%}" if rates else "") + ")")
if len(scored) and len(by_task) and max(len(rs) for rs in by_task.values()) < 3:
print(" NOTE: fewer than 3 runs per task -- enough to show a failure "
"reproduces, not to rank builds.")
if a.out:
with open(a.out, "w") as f:
json.dump({"label": a.label, "base_url": a.base_url,
"model": a.model, "think": cfg.think,
"device": a.device, "repeat": a.repeat,
"passed": n_pass, "total": len(scored),
"skipped": len(rows) - len(scored),
"wall_s": round(wall, 1), "rows": rows}, f, indent=2)
print(f"wrote {a.out}")
if __name__ == "__main__":
main()
@@ -0,0 +1,113 @@
"""Android screens as a text-only agent sees them, with the correct next action.
Each screen is a uiautomator-style accessibility dump flattened to one line per
element -- the same shape droidrun's Portal and AndroidWorld's a11y observation
hand to a text model. `expect` is the action a competent agent must take next;
`accept_names` allows a defensible alternative where one genuinely exists.
"""
SCENARIOS = [
{
"id": "launch_clock",
"task": "Set an alarm for 7:30 AM.",
"screen": """[0] TextView "Phone" clickable=true
[1] TextView "Messages" clickable=true
[2] TextView "Chrome" clickable=true
[3] TextView "Clock" clickable=true
[4] TextView "Settings" clickable=true
[5] TextView "Camera" clickable=true""",
"expect": {"name": "tap", "args": {"index": 3}},
"accept_names": ["tap", "open_app"],
},
{
"id": "clock_alarm_tab",
"task": "Set an alarm for 7:30 AM.",
"screen": """[0] Button "Alarm" clickable=true selected=false
[1] Button "Clock" clickable=true selected=true
[2] Button "Timer" clickable=true selected=false
[3] Button "Stopwatch" clickable=true selected=false
[4] TextView "14:32" clickable=false""",
"expect": {"name": "tap", "args": {"index": 0}},
"accept_names": ["tap"],
},
{
"id": "add_alarm",
"task": "Set an alarm for 7:30 AM.",
"screen": """[0] Button "Alarm" clickable=true selected=true
[1] TextView "No alarms" clickable=false
[2] ImageButton "Add alarm" clickable=true
[3] Button "Timer" clickable=true""",
"expect": {"name": "tap", "args": {"index": 2}},
"accept_names": ["tap"],
},
{
"id": "wifi_settings",
"task": "Turn on Wi-Fi.",
"screen": """[0] TextView "Settings" clickable=false
[1] TextView "Network & internet" clickable=true
[2] TextView "Connected devices" clickable=true
[3] TextView "Apps" clickable=true
[4] TextView "Battery" clickable=true
[5] TextView "Display" clickable=true""",
"expect": {"name": "tap", "args": {"index": 1}},
"accept_names": ["tap"],
},
{
"id": "wifi_toggle",
"task": "Turn on Wi-Fi.",
"screen": """[0] TextView "Network & internet" clickable=false
[1] Switch "Wi-Fi" checked=false clickable=true
[2] TextView "Mobile network" clickable=true
[3] Switch "Airplane mode" checked=false clickable=true""",
"expect": {"name": "tap", "args": {"index": 1}},
"accept_names": ["tap"],
},
{
"id": "search_contact",
"task": "Find the contact named Wei Zhang.",
"screen": """[0] EditText "Search contacts" clickable=true focused=true
[1] TextView "Alice Chen" clickable=true
[2] TextView "Bob Lee" clickable=true
[3] TextView "Carol Wu" clickable=true""",
"expect": {"name": "type_text", "args": {"index": 0, "text": "Wei Zhang"}},
"accept_names": ["type_text"],
},
{
"id": "scroll_for_item",
"task": "Open the Developer options screen in Settings.",
"screen": """[0] TextView "Settings" clickable=false
[1] TextView "Display" clickable=true
[2] TextView "Sound" clickable=true
[3] TextView "Storage" clickable=true
[4] TextView "Privacy" clickable=true
[5] TextView "Location" clickable=true""",
"expect": {"name": "swipe", "args": {"direction": "up"}},
"accept_names": ["swipe"],
},
{
"id": "go_back",
"task": "Return to the previous screen; you opened the wrong menu.",
"screen": """[0] TextView "About phone" clickable=false
[1] TextView "Model: Pixel 8" clickable=false
[2] TextView "Android version" clickable=true""",
"expect": {"name": "press_key", "args": {"key": "back"}},
"accept_names": ["press_key"],
},
{
"id": "open_app_by_name",
"task": "Open the Camera app.",
"screen": """[0] TextView "Swipe up to unlock" clickable=false
[1] TextView "10:04" clickable=false""",
"expect": {"name": "open_app", "args": {"name": "Camera"}},
"accept_names": ["open_app", "swipe"],
},
{
"id": "task_done",
"task": "Turn on Wi-Fi.",
"screen": """[0] TextView "Network & internet" clickable=false
[1] Switch "Wi-Fi" checked=true clickable=true
[2] TextView "Connected to HOME-5G" clickable=false""",
"expect": {"name": "task_complete", "args": {}},
"accept_names": ["task_complete"],
},
]
@@ -0,0 +1,279 @@
"""A scored task suite, judged by what the phone shows rather than by what the
agent said -- or by what the simulator happens to remember.
Three things were wrong with the first version of this file, and all three
made it score more generously than it claimed to:
* **``add_alarm`` passed on any typed string containing "7".** Typing ``7``
and stopping scored a success on a task whose prompt is "Set an alarm for
7:30 AM": no minute, no meridiem, and no requirement that the alarm was ever
saved. It now requires a *committed* alarm reading 7:30 AM, which the clock
screen lists, so the check is one a real phone can also answer.
* **``wifi_on_already`` scored final state only.** Its note said "any tap on
the switch fails the task"; the code could not see a tap, so turning Wi-Fi
off and back on passed. Switch flips are now recorded and the task fails on
any of them -- which is the exact failure W4A8 exhibited (it turned Wi-Fi on
and then tapped the switch off again).
* **Predicates read ``SimulatedDevice`` attributes**, so every scored run was a
simulator run by construction. They read :class:`PhoneState` now, which comes
from the UI dump for both backends.
Repeats are only trials if something differs between them, so each task
carries **variants**: the same goal against a renamed control, a reordered
screen, or a different starting state. Five runs of one fixture at temperature
0 measure repeatability; five variants measure whether the agent can do the
task.
"""
from __future__ import annotations
import random
from dataclasses import dataclass, field
from typing import Callable
from device import SimElement, SimulatedDevice, demo_phone
from phone_state import PhoneState, read_state
Predicate = Callable[[PhoneState], bool]
CLOCK_PACKAGE = "com.google.android.deskclock"
"""Clock app to reset for the alarm tasks. AOSP builds and several OEMs ship a
different package (``com.android.deskclock``, ``com.sec.android.app.clockpackage``
on Samsung); override before a real-device run, because clearing the wrong
package leaves a stale alarm and the next run scores against it."""
# --------------------------------------------------------------- variations
def rename(screen: str, old: str, new: str) -> Callable[[SimulatedDevice], None]:
"""Same control, different word. Android OEMs really do ship "WLAN"."""
def apply(d: SimulatedDevice) -> None:
for el in d.screens[screen].elements:
if el.label == old:
el.label = new
return apply
def reorder(screen: str, seed: int) -> Callable[[SimulatedDevice], None]:
"""Shuffle the tappable rows. Index-based tapping must not be luck."""
def apply(d: SimulatedDevice) -> None:
els = d.screens[screen].elements
head = [e for e in els if not e.clickable]
body = [e for e in els if e.clickable]
random.Random(seed).shuffle(body)
d.screens[screen].elements = head + body
return apply
def decoy(screen: str, label: str) -> Callable[[SimulatedDevice], None]:
"""A plausible wrong row, e.g. "Wi-Fi calling" next to "Wi-Fi"."""
def apply(d: SimulatedDevice) -> None:
d.screens[screen].elements.append(SimElement(label))
return apply
def chain(*fns: Callable[[SimulatedDevice], None]) -> Callable[[SimulatedDevice], None]:
def apply(d: SimulatedDevice) -> None:
for fn in fns:
fn(d)
return apply
@dataclass
class Variant:
variant_id: str
mutate: Callable[[SimulatedDevice], None] = lambda d: None
# --------------------------------------------------------------------- task
@dataclass
class Task:
task_id: str
prompt: str
reaches: Predicate
must_not: Predicate = lambda s: False
setup: Callable[[SimulatedDevice], None] = lambda d: None
max_steps: int = 10
note: str = ""
variants: list[Variant] = field(default_factory=lambda: [Variant("base")])
needs: tuple[str, ...] = ()
"""PhoneState facts this task's predicates require that a screen cannot
show (``toggled``, ``fields``, ``records``). A task listing any of these
cannot be scored on a real phone, and says so instead of scoring wrong."""
reset_adb: tuple[tuple[str, ...], ...] = ()
"""Commands that put a real phone into this task's starting state. Without
them a real-device run is scored against an unknown start, so an empty
tuple means "refuse to run me on a phone"."""
def variant(self, n: int) -> Variant:
return self.variants[n % len(self.variants)]
def build(self, run: int = 0) -> SimulatedDevice:
device = demo_phone()
self.variant(run).mutate(device)
self.setup(device)
return device
def scorable_on(self, state: PhoneState) -> str | None:
"""Why this task cannot be scored from ``state``, or None if it can."""
if state.source == "device" and self.needs:
return (f"needs {', '.join(self.needs)}, which a screen cannot "
f"show; simulator only")
return None
def score(self, device_or_state, declared_done: bool, run: int = 0) -> dict:
state = (device_or_state if isinstance(device_or_state, PhoneState)
else read_state(device_or_state))
blocked = self.scorable_on(state)
if blocked:
return {"task_id": self.task_id, "variant": self.variant(run).variant_id,
"scorable": False, "reason": blocked, "success": False,
"reached_goal": None, "collateral_damage": None,
"declared_done": declared_done}
reached = bool(self.reaches(state))
damaged = bool(self.must_not(state))
return {
"task_id": self.task_id,
"variant": self.variant(run).variant_id,
"scorable": True,
"reached_goal": reached,
"collateral_damage": damaged,
"declared_done": declared_done,
# The only row that counts: the phone is in the requested state,
# nothing else was broken, and the agent knew to stop.
"success": reached and not damaged and declared_done,
}
# ----------------------------------------------------------------- oracles
# Both the on-screen labels and the setting key, because `switch()` matches
# either source: "Wi-Fi" normalises onto the `wifi` setting by luck, but
# "Airplane mode" does not normalise onto `airplane`, and a name the setting
# lookup misses silently falls through to "control not on screen".
WIFI = ("Wi-Fi", "WiFi", "WLAN", "wifi")
AIRPLANE = ("Airplane mode", "Flight mode", "airplane")
def _wifi_is(want: bool) -> Predicate:
# `switch` returns None when the control is not on screen, which is not
# the same as off: an agent that never navigated there has not turned
# anything on.
return lambda s: s.switch(*WIFI) is want
def _airplane_on(s: PhoneState) -> bool:
return s.switch(*AIRPLANE) is True
def _alarm_set(hour: int, minute: int, pm: bool) -> Predicate:
"""A *saved* alarm reading the requested time.
The clock screen lists committed alarms, so this reads the screen and works
on a real phone. Typing the digits without saving leaves the list empty.
"""
wanted = f"{hour}:{minute:02d} {'PM' if pm else 'AM'}"
return lambda s: s.has_text(wanted)
SUITE: list[Task] = [
Task(
task_id="wifi_on",
prompt="Turn on Wi-Fi.",
reaches=_wifi_is(True),
must_not=_airplane_on,
note="Two screens deep, then a switch. The switch's state is visible, "
"so stopping is decidable from the screen.",
variants=[
Variant("base"),
Variant("wlan", rename("network", "Wi-Fi", "WLAN")),
Variant("reordered", reorder("network", seed=7)),
Variant("decoy", decoy("network", "Wi-Fi calling")),
Variant("deep", chain(reorder("settings", seed=3),
decoy("network", "Wi-Fi preferences"))),
],
reset_adb=(("shell", "svc", "wifi", "disable"),),
),
Task(
task_id="wifi_on_already",
prompt="Turn on Wi-Fi.",
setup=lambda d: d.state.__setitem__("wifi", True),
reaches=lambda s: s.switch(*WIFI) is True and s.times_toggled("wifi") == 0,
must_not=_airplane_on,
note="Already satisfied. The only correct action is task_complete. "
"Any flip of the switch fails, including off-and-on again, which "
"final state cannot see and which W4A8 actually did.",
needs=("toggled",),
variants=[
Variant("base"),
Variant("wlan", rename("network", "Wi-Fi", "WLAN")),
Variant("reordered", reorder("network", seed=11)),
],
),
Task(
task_id="wifi_off",
prompt="Turn off Wi-Fi.",
setup=lambda d: d.state.__setitem__("wifi", True),
reaches=_wifi_is(False),
must_not=_airplane_on,
note="The mirror of wifi_on: guards against a model that has simply "
"learned to tap switches.",
variants=[
Variant("base"),
Variant("wlan", rename("network", "Wi-Fi", "WLAN")),
Variant("reordered", reorder("network", seed=5)),
],
reset_adb=(("shell", "svc", "wifi", "enable"),),
),
Task(
task_id="airplane_on",
prompt="Turn on airplane mode.",
reaches=_airplane_on,
note="Same screen as Wi-Fi, different row -- catches index confusion, "
"which is why one variant reorders that screen.",
variants=[
Variant("base"),
Variant("reordered", reorder("network", seed=2)),
Variant("decoy", decoy("network", "Airplane mode settings")),
],
# `settings put global` needs WRITE_SECURE_SETTINGS, which adb has on a
# debuggable build but not on every retail phone; the run fails loudly
# if it is refused rather than scoring against an unknown start.
reset_adb=(("shell", "settings", "put", "global", "airplane_mode_on", "0"),),
),
Task(
task_id="open_clock",
prompt="Open the Clock app and go to the Alarm tab.",
reaches=lambda s: s.has_text("Add alarm") or s.has_text("Set time"),
note="Navigation only, no state change. Isolates 'can it find an app' "
"from 'can it finish'.",
variants=[
Variant("base"),
Variant("reordered", reorder("home", seed=4)),
Variant("decoy", decoy("home", "Clock Widget")),
],
reset_adb=(("shell", "am", "force-stop", CLOCK_PACKAGE),
("shell", "input", "keyevent", "KEYCODE_HOME")),
),
Task(
task_id="add_alarm",
prompt="Set an alarm for 7:30 AM.",
reaches=_alarm_set(7, 30, pm=False),
must_not=lambda s: s.has_text("7:30 PM"),
max_steps=12,
note="The full flow: right app, right tab, add, type both fields, "
"leave the meridiem alone, save. Previously this passed on any "
"typed string containing '7'.",
variants=[
Variant("base"),
Variant("reordered", reorder("clock", seed=9)),
Variant("decoy", decoy("alarms", "Alarm settings")),
],
# `pm clear` removes saved alarms, so run N does not see run N-1's.
reset_adb=(("shell", "pm", "clear", CLOCK_PACKAGE),
("shell", "input", "keyevent", "KEYCODE_HOME")),
),
]
SUITE_BY_ID = {t.task_id: t for t in SUITE}
@@ -0,0 +1,525 @@
"""Tests for the Android agent toolkit.
No phone, emulator or model server is needed: the device is the in-process
simulator and the model is a scripted stand-in. That is deliberate -- this
host has no `adb`, no emulator and no `/dev/kvm`, so anything that could only
be checked against real hardware would otherwise ship unverified.
"""
import json
import pytest
import agent as agent_mod
from agent import AgentConfig, first_tool_call, run_episode
from device import demo_phone
from executor import execute
from ui_tree import parse_ui_dump, render_tree, screen_signature
# --------------------------------------------------------------------- ui_tree
REAL_DUMP = """<?xml version="1.0" encoding="UTF-8"?>
<hierarchy rotation="0">
<node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.android.settings" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2400]">
<node index="1" text="Network &amp; internet" resource-id="android:id/title" class="android.widget.TextView" package="com.android.settings" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,300][1080,460]" />
<node index="2" text="" resource-id="" class="android.widget.LinearLayout" package="com.android.settings" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,460][1080,620]" />
<node index="3" text="" resource-id="com.android.settings:id/switch_widget" class="android.widget.Switch" package="com.android.settings" content-desc="Wi-Fi" checkable="true" checked="true" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[800,460][1000,620]" />
</node>
</hierarchy>"""
def test_parses_a_real_dump_and_keeps_only_what_is_actionable():
els = parse_ui_dump(REAL_DUMP)
labels = [e.text for e in els]
assert "Network & internet" in labels # entity decoded
assert "Wi-Fi" in labels # fell back to content-desc
assert "" not in labels # bare layout node dropped
assert [e.index for e in els] == list(range(len(els)))
def test_switch_state_is_visible_to_the_model():
"""The one clearly wrong action measured earlier was tapping a switch that
was already on; that is only avoidable if the state is on the screen."""
rendered = render_tree(parse_ui_dump(REAL_DUMP))
assert "checked=true" in rendered
assert 'Switch "Wi-Fi"' in rendered
def test_index_maps_to_the_centre_of_the_element():
els = parse_ui_dump(REAL_DUMP)
wifi = next(e for e in els if e.text == "Wi-Fi")
assert wifi.center == (900, 540)
def test_signature_tracks_state_not_pixels():
a = parse_ui_dump(REAL_DUMP)
shifted = REAL_DUMP.replace("[0,300][1080,460]", "[0,310][1080,470]")
assert screen_signature(parse_ui_dump(shifted)) == screen_signature(a)
toggled = REAL_DUMP.replace('checkable="true" checked="true"',
'checkable="true" checked="false"')
assert screen_signature(parse_ui_dump(toggled)) != screen_signature(a)
def test_element_cap_bounds_the_context():
big = '<?xml version="1.0"?><hierarchy>' + "".join(
f'<node text="row{i}" class="android.widget.TextView" clickable="true" '
f'bounds="[0,{i},100,{i + 10}]" />' for i in range(200)
) + "</hierarchy>"
assert len(parse_ui_dump(big, max_elements=25)) == 25
# -------------------------------------------------------------------- executor
def test_tap_lands_on_the_right_row_and_reports_prior_state():
device = demo_phone()
device.current = "network"
els = parse_ui_dump(device.dump())
wifi = next(e for e in els if e.text == "Wi-Fi")
result = execute("tap", {"index": wifi.index}, els, device)
assert result.ok and device.state["wifi"] is True
assert "was off" in result.message
def test_out_of_range_index_is_a_message_not_an_exception():
device = demo_phone()
els = parse_ui_dump(device.dump())
result = execute("tap", {"index": 99}, els, device)
assert not result.ok
assert "out of range" in result.message and f"0-{len(els) - 1}" in result.message
def test_bad_arguments_explain_themselves():
device = demo_phone()
els = parse_ui_dump(device.dump())
assert "missing required argument" in execute("tap", {}, els, device).message
assert "whole number" in execute("tap", {"index": "banana"}, els, device).message
assert "unknown tool" in execute("frobnicate", {}, els, device).message
assert "direction must be" in execute("swipe", {"direction": "sideways"}, els, device).message
def test_type_text_focuses_the_field_first():
device = demo_phone()
device.current = "new_alarm"
els = parse_ui_dump(device.dump())
field = next(e for e in els if e.text == "Hour")
execute("type_text", {"index": field.index, "text": "07"}, els, device)
kinds = [a[0] for a in device.actions]
assert kinds == ["tap", "text"], kinds
assert device.typed == ["07"]
def test_swipe_direction_is_content_movement():
device = demo_phone()
els = parse_ui_dump(device.dump())
execute("swipe", {"direction": "down"}, els, device)
_, x1, y1, x2, y2 = device.actions[-1]
assert int(y1) > int(y2), "scrolling down must move the finger up"
def test_task_complete_carries_its_summary():
device = demo_phone()
result = execute("task_complete", {"summary": "Wi-Fi is on"}, [], device)
assert result.done and result.summary == "Wi-Fi is on"
# ----------------------------------------------------------------------- agent
def _scripted(calls):
"""A stand-in model that plays a fixed list of tool calls."""
seq = list(calls)
def fake_chat(cfg, messages, *, think):
call = seq.pop(0) if seq else None
if call is None:
return {"choices": [{"message": {"content": "", "tool_calls": []}}]}, 0.01
name, args = call
return {"choices": [{"message": {"tool_calls": [
{"function": {"name": name, "arguments": json.dumps(args)}}
]}}]}, 0.01
return fake_chat
def test_first_tool_call_handles_both_argument_encodings():
as_str = {"choices": [{"message": {"tool_calls": [
{"function": {"name": "tap", "arguments": '{"index": 2}'}}]}}]}
as_dict = {"choices": [{"message": {"tool_calls": [
{"function": {"name": "tap", "arguments": {"index": 2}}}]}}]}
assert first_tool_call(as_str) == ("tap", {"index": 2})
assert first_tool_call(as_dict) == ("tap", {"index": 2})
assert first_tool_call({"choices": [{"message": {"content": "hi"}}]}) == (None, {})
# Malformed JSON must not kill the episode.
broken = {"choices": [{"message": {"tool_calls": [
{"function": {"name": "tap", "arguments": "{not json"}}]}}]}
assert first_tool_call(broken) == ("tap", {})
def test_completes_a_multi_screen_task(monkeypatch):
device = demo_phone()
monkeypatch.setattr(agent_mod, "chat", _scripted([
("tap", {"index": 3}), # home -> Settings
("tap", {"index": 1}), # -> Network & internet
("tap", {"index": 1}), # Wi-Fi switch
("task_complete", {"summary": "Wi-Fi enabled"}),
]))
ep = run_episode(device, "Turn on Wi-Fi.", AgentConfig(max_steps=8))
assert ep.done and device.state["wifi"] is True
assert ep.summary == "Wi-Fi enabled"
assert ep.stop_reason == "task_complete"
assert ep.n_steps == 4
def test_says_so_when_an_action_changes_nothing(monkeypatch):
device = demo_phone()
monkeypatch.setattr(agent_mod, "chat", _scripted([
("tap", {"index": 0}), # "Phone" on home: goes nowhere
("task_complete", {"summary": "done"}),
]))
ep = run_episode(device, "Do something.", AgentConfig(max_steps=4))
assert "did not change" in ep.steps[0].result.message
def test_stops_when_it_repeats_a_no_op(monkeypatch):
device = demo_phone()
monkeypatch.setattr(agent_mod, "chat", _scripted([("tap", {"index": 0})] * 6))
ep = run_episode(device, "Do something.", AgentConfig(max_steps=6))
assert not ep.done
assert "stuck" in ep.stop_reason
assert ep.n_steps < 6, "should give up before the budget"
def test_empty_reply_retries_without_thinking(monkeypatch):
"""The measured failure: reasoning never closes `<think>`, so the parser
returns nothing. A non-thinking retry is worth ~0.6 s."""
seen = []
def fake_chat(cfg, messages, *, think):
seen.append(think)
if think:
return {"choices": [{"message": {"content": "", "tool_calls": []}}]}, 8.9
return {"choices": [{"message": {"tool_calls": [
{"function": {"name": "task_complete",
"arguments": '{"summary": "fallback"}'}}]}}]}, 0.61
monkeypatch.setattr(agent_mod, "chat", fake_chat)
ep = run_episode(device := demo_phone(), "Task.", AgentConfig(max_steps=3))
assert seen == [True, False], seen
assert ep.done and ep.summary == "fallback"
assert ep.steps[0].thinking is False
assert device is not None
def test_gives_up_rather_than_looping_when_no_tool_call_ever_comes(monkeypatch):
monkeypatch.setattr(agent_mod, "chat", _scripted([]))
ep = run_episode(demo_phone(), "Task.", AgentConfig(max_steps=5))
assert not ep.done
assert "no tool call" in ep.stop_reason
assert ep.n_steps == 1
def test_step_budget_is_enforced(monkeypatch):
device = demo_phone()
# Alternate two real navigations so nothing is a repeat or a no-op.
monkeypatch.setattr(agent_mod, "chat", _scripted(
[("tap", {"index": 3}), ("press_key", {"key": "back"})] * 5))
ep = run_episode(device, "Wander.", AgentConfig(max_steps=4))
assert not ep.done
assert ep.n_steps == 4
assert "budget exhausted" in ep.stop_reason
def test_transcript_is_readable(monkeypatch):
monkeypatch.setattr(agent_mod, "chat", _scripted([
("tap", {"index": 3}), ("task_complete", {"summary": "ok"})]))
ep = run_episode(demo_phone(), "Open settings.", AgentConfig(max_steps=4))
text = ep.transcript()
assert "task: Open settings." in text
assert "tap(" in text and "task_complete" in text
assert "stopped: task_complete" in text
def test_detects_undoing_its_own_action(monkeypatch):
"""The measured failure: tapping a switch that is already on, turning it
back off, forever. Each tap really does change the screen, so the
no-change check cannot see it."""
device = demo_phone()
device.current = "network"
monkeypatch.setattr(agent_mod, "chat", _scripted([("tap", {"index": 1})] * 6))
ep = run_episode(device, "Turn on Wi-Fi.", AgentConfig(max_steps=6))
assert not ep.done
assert "undoing its own action" in ep.stop_reason
assert any("undid your previous action" in s.result.message
for s in ep.steps if s.result)
def test_normal_navigation_is_not_mistaken_for_oscillation(monkeypatch):
"""Opening a submenu and pressing back revisits a screen; that is how
phones work and must not trip the guard."""
device = demo_phone()
monkeypatch.setattr(agent_mod, "chat", _scripted([
("tap", {"index": 3}), # home -> settings
("press_key", {"key": "back"}), # -> home
("tap", {"index": 2}), # home -> clock
("press_key", {"key": "back"}), # -> home
("task_complete", {"summary": "looked around"}),
]))
ep = run_episode(device, "Look around.", AgentConfig(max_steps=8))
assert ep.done, ep.stop_reason
assert ep.n_steps == 5
# ------------------------------------------------------------------ task suite
def test_suite_scores_on_device_state_not_on_what_the_agent_said():
"""The failure this model exhibits is claiming completion it did not
achieve, so the score must not read the transcript."""
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["wifi_on"]
device = task.build()
assert task.score(device, declared_done=True)["success"] is False
device.state["wifi"] = True
assert task.score(device, declared_done=True)["success"] is True
# Reaching the goal without stopping is not success either.
assert task.score(device, declared_done=False)["success"] is False
def test_collateral_damage_fails_the_task():
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["wifi_on"]
device = task.build()
device.state.update(wifi=True, airplane=True)
score = task.score(device, declared_done=True)
assert score["reached_goal"] and score["collateral_damage"]
assert score["success"] is False
def test_already_satisfied_task_starts_satisfied():
"""Its only correct action is task_complete; tapping the switch fails."""
from phone_state import read_state
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["wifi_on_already"]
device = task.build()
assert task.reaches(read_state(device))
assert task.score(device, declared_done=True)["success"] is True
def test_already_satisfied_task_fails_on_a_switch_round_trip():
"""W4A8 turned Wi-Fi on and then tapped the switch off again. Final state
cannot see that, so this task scored it a pass until the flips were
recorded."""
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["wifi_on_already"]
device = task.build()
device.launch("Settings")
device.tap(540, 360) # Network & internet
device.tap(540, 360) # Wi-Fi off
device.tap(540, 360) # ... and on again
assert device.state["wifi"] is True # final state looks correct
assert task.score(device, declared_done=True)["success"] is False
def test_goal_is_read_from_the_setting_not_from_the_final_screen():
"""An agent that turns Wi-Fi on and then goes Home has still done it."""
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["wifi_on"]
device = task.build()
device.launch("Settings")
device.tap(540, 360)
device.tap(540, 360)
device.keyevent("home")
assert task.score(device, declared_done=True)["success"] is True
def test_never_finding_the_screen_is_not_scored_as_off():
"""`switch()` returns None when neither the setting nor the screen knows
the control; a predicate must not read that as 'off' and pass a task."""
from phone_state import PhoneState
from tasks import SUITE_BY_ID
empty = PhoneState(source="device")
assert empty.switch("Wi-Fi") is None
assert SUITE_BY_ID["wifi_off"].score(empty, declared_done=True)["success"] is False
def test_suite_includes_a_mirror_task_for_switch_tapping():
"""Guards against a model that has simply learned 'tap the switch'."""
from tasks import SUITE_BY_ID
from phone_state import read_state
off = SUITE_BY_ID["wifi_off"]
device = off.build()
assert device.state["wifi"] is True
assert not off.reaches(read_state(device))
device.state["wifi"] = False
assert off.reaches(read_state(device))
def test_chat_retries_transient_failures(monkeypatch):
"""A phone task is a dozen round trips; one dropped connection should cost
a retry, not the episode."""
import agent as mod
from urllib import error as urlerror
calls = {"n": 0}
def flaky(req, timeout=None):
calls["n"] += 1
if calls["n"] < 3:
raise urlerror.URLError("connection reset")
class _R:
def __enter__(self): return self
def __exit__(self, *a): return False
def read(self): return b'{"choices":[{"message":{"tool_calls":[]}}]}'
return _R()
monkeypatch.setattr(mod.urlrequest, "urlopen", flaky)
monkeypatch.setattr(mod.time, "sleep", lambda s: None)
cfg = AgentConfig(retries=3)
body, _ = mod.chat(cfg, [{"role": "user", "content": "x"}], think=False)
assert calls["n"] == 3 and "choices" in body
def test_chat_gives_up_with_a_clear_error(monkeypatch):
import agent as mod
from urllib import error as urlerror
monkeypatch.setattr(mod.urlrequest, "urlopen",
lambda *a, **k: (_ for _ in ()).throw(urlerror.URLError("down")))
monkeypatch.setattr(mod.time, "sleep", lambda s: None)
with pytest.raises(mod.ChatError, match="after 2 attempts"):
mod.chat(AgentConfig(retries=2), [{"role": "user", "content": "x"}], think=False)
# --------------------------------------------------- oracles that were wrong
def test_alarm_task_needs_a_saved_alarm_not_a_typed_digit():
"""This oracle used to pass on any typed string containing "7", so typing
`7` and stopping scored a success on "Set an alarm for 7:30 AM"."""
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["add_alarm"]
typed_only = task.build()
typed_only.launch("Clock")
typed_only.tap(540, 200) # Alarm tab
typed_only.tap(540, 360) # Add alarm
typed_only.tap(540, 360)
typed_only.input_text("7")
assert task.score(typed_only, declared_done=True)["success"] is False
both_fields = task.build()
both_fields.launch("Clock")
both_fields.tap(540, 200)
both_fields.tap(540, 360)
both_fields.tap(540, 360)
both_fields.input_text("7")
both_fields.tap(540, 520)
both_fields.input_text("30")
assert task.score(both_fields, declared_done=True)["success"] is False, \
"typing both fields without saving is not a set alarm"
both_fields.tap(540, 840) # Save
assert task.score(both_fields, declared_done=True)["success"] is True
def test_alarm_task_rejects_the_wrong_meridiem():
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["add_alarm"]
d = task.build()
d.launch("Clock"); d.tap(540, 200); d.tap(540, 360)
d.tap(540, 360); d.input_text("7")
d.tap(540, 520); d.input_text("30")
d.tap(540, 680) # AM -> PM
d.tap(540, 840) # Save
score = task.score(d, declared_done=True)
assert score["success"] is False and score["collateral_damage"] is True
def test_a_saved_alarm_is_visible_on_the_screen():
"""The oracle reads the clock app's list, so it works on a real phone."""
from phone_state import read_state
from tasks import SUITE_BY_ID
d = SUITE_BY_ID["add_alarm"].build()
d.launch("Clock"); d.tap(540, 200); d.tap(540, 360)
d.tap(540, 360); d.input_text("7")
d.tap(540, 520); d.input_text("30")
d.tap(540, 840)
assert read_state(d).has_text("7:30 AM")
# ------------------------------------------------------------------ variants
def test_every_task_has_variants_so_repeats_are_trials():
"""Five runs of one fixture at temperature 0 measure repeatability. The
suite has to vary something for a repeat to be a trial."""
from tasks import SUITE
for task in SUITE:
assert len(task.variants) >= 1
ids = [v.variant_id for v in task.variants]
assert len(ids) == len(set(ids)), task.task_id
assert any(len(t.variants) > 1 for t in SUITE)
def test_variants_change_the_phone_but_not_the_goal():
"""A renamed or reordered control is still the same task."""
from phone_state import read_state
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["wifi_on"]
labels_seen = set()
for run in range(len(task.variants)):
d = task.build(run)
d.launch("Settings")
d.tap(540, 360)
labels_seen.add(read_state(d).source)
# The goal predicate is unchanged; only the route differs.
assert task.score(d, declared_done=True)["success"] is False
d.state["wifi"] = True
assert task.score(d, declared_done=True)["success"] is True
assert labels_seen == {"sim"}
def test_wlan_variant_renames_the_control_the_agent_must_find():
from phone_state import read_state
from tasks import SUITE_BY_ID
task = SUITE_BY_ID["wifi_on"]
variant = next(i for i, v in enumerate(task.variants) if v.variant_id == "wlan")
d = task.build(variant)
d.launch("Settings"); d.tap(540, 360)
assert read_state(d).has_text("WLAN")
assert not read_state(d).has_text("Wi-Fi")
# -------------------------------------------------- real device honesty
def test_history_dependent_task_refuses_to_score_on_a_real_phone():
"""A phone cannot report which switches were flipped, so the task says so
instead of scoring from final state and calling a round trip a pass."""
from phone_state import PhoneState
from tasks import SUITE_BY_ID
state = PhoneState(source="device", settings={"wifi": True})
score = SUITE_BY_ID["wifi_on_already"].score(state, declared_done=True)
assert score["scorable"] is False
assert "simulator only" in score["reason"]
assert score["success"] is False
def test_tasks_runnable_on_a_phone_declare_how_to_reset_it():
"""Without a reset the run is scored against an unknown starting state."""
from tasks import SUITE_BY_ID
assert SUITE_BY_ID["wifi_on"].reset_adb
assert SUITE_BY_ID["wifi_off"].reset_adb
# and the sim-only one is not pretending otherwise
assert SUITE_BY_ID["wifi_on_already"].needs == ("toggled",)
@@ -0,0 +1,155 @@
"""Recover a tool call from what the model actually emitted.
vLLM's stock `glm45` parser handles Spark's tool output most of the time --
4 of 5 screens in a spot check -- and then does not. On the Wi-Fi toggle
screen, the one where a switch's state decides the right action, the model
dropped out of GLM-4.5 format entirely and emitted
[tap]
{"tool": "tap", "parameters": {"index": 1}
with the JSON unterminated. `tool_calls` comes back empty, and an agent that
only reads `tool_calls` sees a step with no action at all.
For a phone agent that is the difference between finishing a task and
stopping in the middle of it, so this module tries, in order:
1. the parsed `tool_calls` (the normal path -- nothing here second-guesses it),
2. a GLM-4.5 `<tool_call>` block left in the content,
3. the bracketed `[name]` + JSON form above, repairing unbalanced braces,
4. a bare JSON object naming a tool.
Every recovery is reported, because an agent that quietly repairs its model's
output will hide a regression in that model.
"""
from __future__ import annotations
import json
import re
from dataclasses import dataclass
_BRACKET = re.compile(r"\[([a-z_][a-z0-9_]*)\]", re.I)
_GLM_CALL = re.compile(
r"<tool_call>\s*([a-z_][a-z0-9_]*)(.*?)</tool_call>", re.I | re.S)
_GLM_ARG = re.compile(
r"<arg_key>(.*?)</arg_key>\s*<arg_value>(.*?)</arg_value>", re.I | re.S)
_ARG_CONTAINERS = ("parameters", "arguments", "args")
_NAME_KEYS = ("tool", "name", "tool_name", "function")
@dataclass
class ParsedCall:
name: str | None
args: dict
source: str
"""Which path produced it: tool_calls | glm_block | bracketed | bare_json."""
@property
def recovered(self) -> bool:
return self.name is not None and self.source != "tool_calls"
def _coerce(value: str):
"""Tool arguments arrive as strings; indices must become integers."""
text = value.strip()
if re.fullmatch(r"-?\d+", text):
return int(text)
if text.lower() in ("true", "false"):
return text.lower() == "true"
return text
def _balance(text: str) -> str:
"""Close a JSON object the model stopped short of finishing."""
depth_curly = text.count("{") - text.count("}")
depth_square = text.count("[") - text.count("]")
if depth_curly < 0 or depth_square < 0:
return text
return text + "]" * depth_square + "}" * depth_curly
def _load(text: str) -> dict | None:
for candidate in (text, _balance(text)):
try:
value = json.loads(candidate)
except json.JSONDecodeError:
continue
if isinstance(value, dict):
return value
return None
def _unwrap(payload: dict, fallback_name: str | None) -> tuple[str | None, dict]:
"""Pull a (name, args) pair out of assorted self-describing shapes."""
name = fallback_name
for key in _NAME_KEYS:
value = payload.get(key)
if isinstance(value, str) and value:
name = value
break
if isinstance(value, dict) and isinstance(value.get("name"), str):
name = value["name"]
payload = {**payload, **value}
break
args: dict = {}
for key in _ARG_CONTAINERS:
inner = payload.get(key)
if isinstance(inner, dict):
args = dict(inner)
break
if isinstance(inner, str):
parsed = _load(inner)
if parsed is not None:
args = parsed
break
if not args:
args = {k: v for k, v in payload.items()
if k not in _NAME_KEYS + _ARG_CONTAINERS}
return name, args
def _first_json_object(text: str) -> str | None:
start = text.find("{")
return text[start:] if start >= 0 else None
def extract_tool_call(message: dict) -> ParsedCall:
"""The one call to act on, from whichever shape the model used."""
calls = message.get("tool_calls") or []
if calls:
fn = calls[0].get("function", {}) or {}
raw = fn.get("arguments")
if isinstance(raw, dict):
args = raw
else:
args = _load(raw or "{}") or {}
if set(args) & set(_ARG_CONTAINERS):
_, args = _unwrap(args, fn.get("name"))
return ParsedCall(fn.get("name"), args, "tool_calls")
content = message.get("content") or ""
reasoning = message.get("reasoning_content") or ""
for text in (content, reasoning):
if not text.strip():
continue
if m := _GLM_CALL.search(text):
args = {k.strip(): _coerce(v) for k, v in _GLM_ARG.findall(m.group(2))}
return ParsedCall(m.group(1), args, "glm_block")
bracket = _BRACKET.search(text)
blob = _first_json_object(text)
if blob is not None:
payload = _load(blob)
if payload is not None:
name, args = _unwrap(payload, bracket.group(1) if bracket else None)
if name:
return ParsedCall(name, args, "bracketed" if bracket else "bare_json")
if bracket and not blob:
# A naked "[press_key]" with no arguments is still an instruction.
return ParsedCall(bracket.group(1), {}, "bracketed")
return ParsedCall(None, {}, "none")
@@ -0,0 +1,146 @@
"""Turn a uiautomator dump into what a text-only agent can act on.
Spark-X2.5 never sees a screenshot, so the whole agent hinges on this file:
a screen has to arrive as a short, stable, indexed list of elements, and an
index the model returns has to map back to a point on the glass.
Two decisions shape the format, both forced by the model rather than by taste:
* **Filter hard.** A real settings screen dumps 150+ nodes, most of them
layout containers. At 56 KiB/token of KV cache and a 1.7B model that both
costs context and buries the answer, so only nodes that are interactive or
carry text survive.
* **Index, do not coordinate.** The model is asked for `index: 3`, never for
pixels -- a text model has no way to know where 3 is on the glass, and
coordinates in the transcript would be a source of silent nonsense. The
mapping back to a centre point happens here, where the bounds are known.
The rendered form is the one the scenario fixtures were written in, so the
decision benchmark and the live agent observe identically.
"""
from __future__ import annotations
import re
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
_BOUNDS = re.compile(r"\[(-?\d+),(-?\d+)\]\[(-?\d+),(-?\d+)\]")
# Attributes worth showing when they are true, in the order a reader wants
# them. `checked` earns its place: the one clearly wrong action in the earlier
# measurements was tapping a Wi-Fi switch that was already on, which is only
# avoidable if the state is on the screen the model sees.
_FLAGS = ("clickable", "checked", "selected", "scrollable", "focused")
@dataclass
class Element:
index: int
cls: str
text: str
resource_id: str = ""
package: str = ""
bounds: tuple[int, int, int, int] = (0, 0, 0, 0)
flags: dict[str, bool] = field(default_factory=dict)
editable: bool = False
@property
def center(self) -> tuple[int, int]:
x1, y1, x2, y2 = self.bounds
return (x1 + x2) // 2, (y1 + y2) // 2
@property
def short_class(self) -> str:
return self.cls.rsplit(".", 1)[-1] if self.cls else "View"
def render(self) -> str:
parts = [f"[{self.index}]", self.short_class, f'"{self.text}"']
for flag in _FLAGS:
if flag in self.flags:
parts.append(f"{flag}={str(self.flags[flag]).lower()}")
return " ".join(parts)
def _truthy(node: ET.Element, name: str) -> bool:
return node.get(name, "false") == "true"
def _label(node: ET.Element) -> str:
"""What to call this element, preferring what a person would read."""
for attr in ("text", "content-desc"):
value = (node.get(attr) or "").strip()
if value:
return value.replace("\n", " ")
# A bare resource id is a poor label but far better than an empty string
# when the element is the only way forward (unlabelled icon buttons).
rid = node.get("resource-id") or ""
return rid.rsplit("/", 1)[-1] if rid else ""
def _interesting(node: ET.Element) -> bool:
if any(_truthy(node, a) for a in ("clickable", "checkable", "scrollable",
"long-clickable", "focusable")):
return True
return bool(_label(node)) and not _truthy(node, "password")
def parse_bounds(raw: str | None) -> tuple[int, int, int, int]:
m = _BOUNDS.search(raw or "")
if not m:
return (0, 0, 0, 0)
return tuple(int(g) for g in m.groups()) # type: ignore[return-value]
def parse_ui_dump(xml_text: str, *, max_elements: int = 60) -> list[Element]:
"""uiautomator XML -> the elements worth showing, in reading order."""
root = ET.fromstring(xml_text)
elements: list[Element] = []
for node in root.iter("node"):
if not _interesting(node):
continue
label = _label(node)
clickable = _truthy(node, "clickable")
# An unlabelled, non-interactive node tells the model nothing it can
# use and costs it context.
if not label and not clickable and not _truthy(node, "checkable"):
continue
flags = {"clickable": clickable}
if _truthy(node, "checkable"):
flags["checked"] = _truthy(node, "checked")
if _truthy(node, "selected"):
flags["selected"] = True
if _truthy(node, "scrollable"):
flags["scrollable"] = True
if _truthy(node, "focused"):
flags["focused"] = True
elements.append(
Element(
index=len(elements),
cls=node.get("class", ""),
text=label,
resource_id=node.get("resource-id", ""),
package=node.get("package", ""),
bounds=parse_bounds(node.get("bounds")),
flags=flags,
editable="EditText" in (node.get("class") or ""),
)
)
if len(elements) >= max_elements:
break
return elements
def render_tree(elements: list[Element]) -> str:
"""The exact text the model is shown."""
return "\n".join(e.render() for e in elements)
def screen_signature(elements: list[Element]) -> str:
"""A cheap identity for a screen, to notice when an action changed nothing.
Labels and flags only: coordinates shift with animations and scroll offsets,
and treating those as a new screen would hide a stuck agent.
"""
return "|".join(f"{e.short_class}:{e.text}:{e.flags.get('checked', '')}"
for e in elements)