Package this fork as vllm-embedding: README, docs, roadmap, harness
Turns a branch of accumulated edge/CPU work into something someone else can
read, reproduce and continue.
README.md states what the fork is for (multimodal models on edge CPUs rather
than servers), the headline memory and speed results with their costs, and why
the numbers are worth reading -- five retracted conclusions, all measurement
errors rather than coding errors. Upstream's README is preserved at
docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the
files this fork changes.
docs/edge/:
methodology.md the measurement discipline, which is the most transferable
thing here: what each of the five retractions was, and the
rule it left behind. Includes the two traps specific to this
stack -- vLLM's AOT compile cache keying on config so two
arms sharing an artifact load the wrong graph, and allocated
not being resident until mincore(2) says so.
results.md every measured figure with its spread and baseline, and an
explicit note that the four memory levers overlap and must
not be added.
cpu-4bit.md the two 4-bit paths, the packed layout as measured rather
than assumed, why the path choice is a config field, and the
GPTQ output-head trap that invalidated a whole round of W4A8
numbers.
harness.md how to run a comparison, and the two arguments that matter
more than they look (--context-len, --prompt-len), each of
which has produced a wrong conclusion by being left at its
default.
ROADMAP.md orders by whether the verification method is correct rather than by
expected size, lists what is done with results, what is next with expected
gains and how each would be verified, and -- at equal length -- what was
considered and rejected, including the CPU megakernel with the measurements
that rule it out for now.
benchmarks/edge_harness/ brings the tooling into the repo so the numbers are
reproducible: the grid runner, the single-run benchmarks, the memory ledger,
the fidelity metric, the Android agent suite, the quantizers, and the tests
that pin the harness's judgement. 82 tests pass from the new location.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This commit is contained in:
1 parent
289c13df68
commit
b36a3431bd
32 files changed
+5641
-86
No files matched your search
@@ -0,0 +1,92 @@
|
||||
# Android agent toolkit for Spark-X2.5
|
||||
|
||||
Lets a text-only model operate a phone: observe the accessibility tree, call
|
||||
one tool, execute it on the device, observe again.
|
||||
|
||||
ui_tree.py uiautomator XML -> indexed element list -> back to a tap point
|
||||
device.py AdbDevice (a real phone) | SimulatedDevice (scripted, in-process)
|
||||
executor.py one tool call -> one device action, errors as sentences
|
||||
tool_parse.py recover a call from whatever shape the model emitted
|
||||
agent.py the loop: budget, no-op detection, undo detection, fallbacks
|
||||
run_episode.py CLI
|
||||
|
||||
## Running it
|
||||
|
||||
Serve the model (Spark emits GLM-4.5 tool format, so the stock parsers apply):
|
||||
|
||||
vllm serve <model> --port 8077 --enable-auto-tool-choice \
|
||||
--tool-call-parser glm45 --reasoning-parser glm45
|
||||
|
||||
Then:
|
||||
|
||||
python run_episode.py --task "Turn on Wi-Fi." --device sim # no phone needed
|
||||
python run_episode.py --task "Turn on Wi-Fi." --device adb # a real phone
|
||||
|
||||
`pytest test_android_agent.py` runs 21 tests with no phone and no server.
|
||||
|
||||
## What the loop does that a plain ReAct loop does not
|
||||
|
||||
Each one is a response to a measured failure of *this* model, not generic
|
||||
scaffolding:
|
||||
|
||||
- **Retry without thinking on an empty reply.** The model can reason past its
|
||||
token budget without closing `<think>`; vLLM's reasoning parser only emits
|
||||
content once the block closes, so the reply arrives empty with no tool call.
|
||||
Raising the cap buys a longer silence. A non-thinking retry costs ~0.6 s.
|
||||
- **Report actions that changed nothing.** "the screen did not change" is
|
||||
information the model cannot otherwise have.
|
||||
- **Report and stop on self-undo.** Tapping a switch that is already on undoes
|
||||
the task. Each tap really does change the screen, so a no-change check
|
||||
cannot see it; the loop watches for the *same* action returning the screen to
|
||||
its previous state. Deliberately narrow -- opening a submenu and pressing
|
||||
back revisits a screen and must not trip it.
|
||||
- **Recover malformed tool calls.** See below.
|
||||
- **A step budget**, so a confused episode ends instead of running forever.
|
||||
|
||||
The loop never decides a task is finished. It reports what happened and lets
|
||||
the model call `task_complete`; a harness that completes tasks on the model's
|
||||
behalf measures itself.
|
||||
|
||||
## Measured: which build can actually finish a task
|
||||
|
||||
Simulated phone, one run per cell, `max_steps` 10, Xeon 8480C 24 threads.
|
||||
|
||||
| build | decode | "Turn on Wi-Fi." | "Set an alarm for 7:30 AM." |
|
||||
|---|---|---|---|
|
||||
| bf16 | 41.0 tok/s | done in 4 steps | wrong app (Settings, not Clock), toggles Airplane mode, stuck |
|
||||
| **W4A16 g128** | 80.0 tok/s | done in 4 steps | right app, full navigation (Clock -> Alarm -> Add -> Save), then no tool call |
|
||||
| W4A8 g128 | 86.6 tok/s | turns Wi-Fi on, then toggles it off again (2 runs, same failure) | taps "Phone" twice, reaches Alarm, then loops |
|
||||
|
||||
Two things fall out, and both matter more than the throughput numbers:
|
||||
|
||||
**Navigation survives quantization; completion does not.** Every build walks
|
||||
home -> Settings -> Network -> Wi-Fi correctly. Only some of them notice that
|
||||
the switch now reads `checked=true` and stop. W4A8 -- the fastest build, and
|
||||
the one the speed-first selector recommends -- turns Wi-Fi on and then taps it
|
||||
off again, ending with the task undone.
|
||||
|
||||
**The fidelity number had a task-level meaning all along.** W4A8 reproduced
|
||||
1/12 greedy prompts against its own bf16 and W4A16 2/12
|
||||
(analysis/spark_edge_support.md 2c). That looked like a small difference on a
|
||||
proxy metric. It is the difference between finishing and not.
|
||||
|
||||
So for agentic use the recommendation is **W4A16**, not W4A8: 8% slower, and
|
||||
it is the only 4-bit build here that completed anything. `weight_path.py`'s
|
||||
`priority="fidelity"` selects it.
|
||||
|
||||
## Honest limits
|
||||
|
||||
- **No phone was involved.** This host has no `adb`, no emulator and no
|
||||
`/dev/kvm`. The simulator emits real uiautomator XML, so the parser, the
|
||||
index-to-coordinate mapping and the stopping rules are exercised on the path
|
||||
hardware drives -- but whether a real Settings app is laid out the way these
|
||||
fixtures assume is untested.
|
||||
- **Two tasks, one run each.** Enough to show a failure mode reproduces, not
|
||||
enough to rank builds by success rate.
|
||||
- **Stock `glm45` does not always parse Spark.** On 4 of 5 screens it does. On
|
||||
the Wi-Fi toggle screen the model emitted
|
||||
`[tap]\n{"tool": "tap", "parameters": {"index": 1}` -- unterminated, and
|
||||
`tool_calls` came back empty. `tool_parse.py` recovers that and reports every
|
||||
recovery, because an agent that silently repairs its model's output hides
|
||||
regressions in that model. The earlier claim that the stock parsers work
|
||||
"as-is" is true most of the time, which is not the same thing.
|
||||
@@ -0,0 +1,152 @@
|
||||
"""Android control action space for a text-only LLM agent.
|
||||
|
||||
The tool surface mirrors what the mainstream ADB-based harnesses expose
|
||||
(droidrun/mobilerun's Portal tools, ghost-in-the-droid's touch/see tools,
|
||||
AndroidWorld's `JSONAction`), so an agent written against these schemas can be
|
||||
pointed at any of them. Every action is expressed over the accessibility tree
|
||||
rather than pixel coordinates, because Spark-X2.5 is a text-only model: it
|
||||
never sees a screenshot, it sees the uiautomator dump.
|
||||
|
||||
`ADB_TEMPLATES` gives the literal `adb` invocation each tool maps to, which is
|
||||
what a driver executes against a real device.
|
||||
"""
|
||||
|
||||
TOOLS = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "tap",
|
||||
"description": (
|
||||
"Tap a UI element. Use the element's index exactly as shown in "
|
||||
"the accessibility tree."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"index": {
|
||||
"type": "integer",
|
||||
"description": "Index of the element in the tree",
|
||||
}
|
||||
},
|
||||
"required": ["index"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "type_text",
|
||||
"description": (
|
||||
"Type text into a focused input field. Tap the field first."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"index": {"type": "integer", "description": "Index of the input field"},
|
||||
"text": {"type": "string", "description": "Text to enter"},
|
||||
},
|
||||
"required": ["index", "text"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "swipe",
|
||||
"description": (
|
||||
"Scroll the screen. The direction is where the content should "
|
||||
"move: use 'down' to see what is further down the page."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"direction": {
|
||||
"type": "string",
|
||||
"enum": ["up", "down", "left", "right"],
|
||||
}
|
||||
},
|
||||
"required": ["direction"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "press_key",
|
||||
"description": "Press a hardware or system key.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"key": {
|
||||
"type": "string",
|
||||
"enum": ["back", "home", "enter", "delete", "recent"],
|
||||
}
|
||||
},
|
||||
"required": ["key"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "open_app",
|
||||
"description": "Launch an app by its visible name.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"name": {"type": "string"}},
|
||||
"required": ["name"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "task_complete",
|
||||
"description": (
|
||||
"Call when the user's request has been fully carried out."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"summary": {"type": "string", "description": "What was done"}
|
||||
},
|
||||
"required": ["summary"],
|
||||
},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
# How each tool becomes a real device command. `{bounds}` is filled from the
|
||||
# element's centre point in the accessibility tree.
|
||||
ADB_TEMPLATES = {
|
||||
"tap": "adb shell input tap {x} {y}",
|
||||
"type_text": "adb shell input text {text_escaped}",
|
||||
"swipe": "adb shell input swipe {x1} {y1} {x2} {y2} 300",
|
||||
"press_key": "adb shell input keyevent {keycode}",
|
||||
"open_app": "adb shell monkey -p {package} -c android.intent.category.LAUNCHER 1",
|
||||
}
|
||||
|
||||
KEYCODES = {
|
||||
"back": "KEYCODE_BACK",
|
||||
"home": "KEYCODE_HOME",
|
||||
"enter": "KEYCODE_ENTER",
|
||||
"delete": "KEYCODE_DEL",
|
||||
"recent": "KEYCODE_APP_SWITCH",
|
||||
}
|
||||
|
||||
SYSTEM_PROMPT = (
|
||||
"You control an Android phone. At each step you are given the current "
|
||||
"screen as an accessibility tree: one line per element, with its index, "
|
||||
"type, text and whether it is clickable. Call exactly one tool per step to "
|
||||
"make progress on the user's task. Refer to elements only by the index "
|
||||
"shown.\n"
|
||||
# Both rules below exist because the model failed them in measurement: it
|
||||
# tapped a Wi-Fi switch that already read checked=true, turning it back
|
||||
# off, and never called task_complete. A switch's state is on the screen,
|
||||
# so the rule can be stated in terms of what it can see.
|
||||
"A switch or checkbox shows its state as checked=true or checked=false. "
|
||||
"Tapping it flips that state.\n"
|
||||
"Before acting, check whether the screen already shows what the task "
|
||||
"asked for. If it does, call task_complete immediately and do not tap "
|
||||
"anything else -- acting again would undo it."
|
||||
)
|
||||
@@ -0,0 +1,271 @@
|
||||
"""The observe-decide-act loop.
|
||||
|
||||
Three things in here are not generic agent scaffolding; they are direct
|
||||
responses to how this model failed when it was measured one step at a time
|
||||
(analysis/spark_edge_support.md section 5):
|
||||
|
||||
* **A step budget, not a bigger token cap.** On the one genuinely ambiguous
|
||||
screen the model reasoned past 1200 tokens without closing `<think>`, and
|
||||
vLLM's reasoning parser only emits content once the block closes -- so the
|
||||
reply came back empty, with no tool call at all. Raising the cap buys a
|
||||
longer silence. The loop instead retries that step with thinking disabled,
|
||||
which measured 0.61 s against 8.9 s, and takes the weaker-but-present answer.
|
||||
|
||||
* **Detecting the no-op.** The clearest wrong action was tapping a Wi-Fi
|
||||
switch that was already on -- undoing the task it had just completed. An
|
||||
action that leaves the screen signature unchanged is reported back as such,
|
||||
because "nothing happened" is information the model cannot otherwise have.
|
||||
|
||||
* **Repeat suppression.** Two identical actions on an identical screen means
|
||||
stuck, and a third will not help. The loop says so in the transcript and
|
||||
stops, rather than burning the budget.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from urllib import error as urlerror
|
||||
from urllib import request as urlrequest
|
||||
|
||||
from action_space import SYSTEM_PROMPT, TOOLS
|
||||
from device import Device
|
||||
from executor import ActionResult, execute
|
||||
from tool_parse import extract_tool_call
|
||||
from ui_tree import Element, parse_ui_dump, render_tree, screen_signature
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentConfig:
|
||||
base_url: str = "http://127.0.0.1:8077/v1"
|
||||
model: str = "XHToken/Spark-X2.5-1.7B"
|
||||
max_steps: int = 12
|
||||
think: bool = True
|
||||
think_fallback: bool = True
|
||||
"""On an empty reply, retry the step without thinking rather than fail."""
|
||||
max_tokens: int = 1024
|
||||
temperature: float = 0.0
|
||||
timeout: float = 900.0
|
||||
history: int = 6
|
||||
retries: int = 3
|
||||
"""Transient HTTP failures retried before a step is abandoned."""
|
||||
retry_backoff_s: float = 2.0
|
||||
"""Recent actions shown back to the model."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Step:
|
||||
n: int
|
||||
screen: str
|
||||
tool: str | None
|
||||
args: dict
|
||||
result: ActionResult | None
|
||||
latency_s: float
|
||||
thinking: bool
|
||||
note: str = ""
|
||||
parsed_via: str = "tool_calls"
|
||||
"""Which parser recovered the call; anything but tool_calls is worth seeing."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Episode:
|
||||
task: str
|
||||
steps: list[Step] = field(default_factory=list)
|
||||
done: bool = False
|
||||
summary: str = ""
|
||||
stop_reason: str = ""
|
||||
|
||||
@property
|
||||
def n_steps(self) -> int:
|
||||
return len(self.steps)
|
||||
|
||||
def transcript(self) -> str:
|
||||
lines = [f"task: {self.task}"]
|
||||
for s in self.steps:
|
||||
call = f"{s.tool}({json.dumps(s.args, ensure_ascii=False)})" if s.tool else "(no tool call)"
|
||||
outcome = s.result.message if s.result else s.note
|
||||
if s.parsed_via not in ("tool_calls", "none"):
|
||||
outcome += f" [recovered via {s.parsed_via}]"
|
||||
lines.append(f" {s.n:2d}. {call:48s} -> {outcome} [{s.latency_s:.2f}s]")
|
||||
lines.append(f"stopped: {self.stop_reason}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
class ChatError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def chat(cfg: AgentConfig, messages: list[dict], *, think: bool) -> tuple[dict, float]:
|
||||
payload = {
|
||||
"model": cfg.model,
|
||||
"messages": messages,
|
||||
"tools": TOOLS,
|
||||
"tool_choice": "auto",
|
||||
"temperature": cfg.temperature,
|
||||
"max_tokens": cfg.max_tokens,
|
||||
}
|
||||
if not think:
|
||||
# Spark's chat template opens <think> by default; closing it up front
|
||||
# is what makes the model answer directly.
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": False}
|
||||
req = urlrequest.Request(
|
||||
f"{cfg.base_url}/chat/completions",
|
||||
data=json.dumps(payload).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
t0 = time.perf_counter()
|
||||
last: Exception | None = None
|
||||
for attempt in range(max(1, cfg.retries)):
|
||||
try:
|
||||
with urlrequest.urlopen(req, timeout=cfg.timeout) as r:
|
||||
return json.loads(r.read()), time.perf_counter() - t0
|
||||
except (urlerror.URLError, TimeoutError, json.JSONDecodeError) as exc:
|
||||
# A phone task is a dozen round trips; one dropped connection
|
||||
# should cost a retry, not the episode.
|
||||
last = exc
|
||||
if attempt + 1 < max(1, cfg.retries):
|
||||
time.sleep(cfg.retry_backoff_s * (attempt + 1))
|
||||
raise ChatError(
|
||||
f"{cfg.base_url} failed after {cfg.retries} attempts: {last}") from last
|
||||
|
||||
|
||||
def wait_for_server(cfg: AgentConfig, timeout_s: float = 300.0) -> bool:
|
||||
"""Block until the endpoint serves a model list, or give up.
|
||||
|
||||
Starting an episode against a server that is still loading produces a
|
||||
confusing transcript full of connection errors; this turns that into one
|
||||
clear wait.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with urlrequest.urlopen(f"{cfg.base_url}/models", timeout=5) as r:
|
||||
if b'"id"' in r.read():
|
||||
return True
|
||||
except Exception:
|
||||
time.sleep(2.0)
|
||||
return False
|
||||
|
||||
|
||||
def parse_response(response: dict):
|
||||
"""The one call to act on, from whichever shape the model used.
|
||||
|
||||
Spark does not always stay in GLM-4.5 tool format -- see tool_parse -- and
|
||||
on a phone a dropped step is a stalled task, so the fallbacks matter.
|
||||
"""
|
||||
try:
|
||||
message = response["choices"][0]["message"]
|
||||
except (KeyError, IndexError):
|
||||
message = {}
|
||||
return extract_tool_call(message)
|
||||
|
||||
|
||||
def first_tool_call(response: dict) -> tuple[str | None, dict]:
|
||||
"""Name and arguments only, for callers that do not care how it parsed."""
|
||||
parsed = parse_response(response)
|
||||
return parsed.name, parsed.args
|
||||
|
||||
|
||||
def _user_message(task: str, screen: str, log: list[str], history: int) -> str:
|
||||
parts = [f"Task: {task}", "", "Current screen:", screen]
|
||||
if log:
|
||||
parts += ["", "Actions so far:"] + [f" {line}" for line in log[-history:]]
|
||||
parts += ["", "Choose the next action. Call exactly one tool."]
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def observe(device: Device) -> tuple[list[Element], str, str]:
|
||||
elements = parse_ui_dump(device.dump())
|
||||
return elements, render_tree(elements), screen_signature(elements)
|
||||
|
||||
|
||||
def run_episode(device: Device, task: str, cfg: AgentConfig | None = None) -> Episode:
|
||||
cfg = cfg or AgentConfig()
|
||||
episode = Episode(task=task)
|
||||
log: list[str] = []
|
||||
recent: list[tuple[str, str]] = [] # (action, screen signature before it)
|
||||
signature_before_last: str | None = None
|
||||
last_action: str | None = None
|
||||
oscillations = 0
|
||||
|
||||
for n in range(1, cfg.max_steps + 1):
|
||||
elements, screen, signature = observe(device)
|
||||
messages = [
|
||||
{"role": "system", "content": SYSTEM_PROMPT},
|
||||
{"role": "user",
|
||||
"content": _user_message(task, screen, log, cfg.history)},
|
||||
]
|
||||
|
||||
thinking = cfg.think
|
||||
response, latency = chat(cfg, messages, think=thinking)
|
||||
parsed = parse_response(response)
|
||||
tool, args = parsed.name, parsed.args
|
||||
|
||||
if tool is None and cfg.think_fallback and thinking:
|
||||
# The measured failure: reasoning never closed, so the parser
|
||||
# emitted nothing. A second, non-thinking pass is worth ~0.6 s.
|
||||
thinking = False
|
||||
retry, retry_latency = chat(cfg, messages, think=False)
|
||||
latency += retry_latency
|
||||
parsed = parse_response(retry)
|
||||
tool, args = parsed.name, parsed.args
|
||||
|
||||
if tool is None:
|
||||
episode.steps.append(Step(n, screen, None, {}, None, latency, thinking,
|
||||
note="model returned no tool call",
|
||||
parsed_via=parsed.source))
|
||||
episode.stop_reason = "no tool call, even without thinking"
|
||||
return episode
|
||||
|
||||
if (tool_key := f"{tool}:{json.dumps(args, sort_keys=True)}") and \
|
||||
recent.count((tool_key, signature)) >= 2:
|
||||
episode.steps.append(Step(n, screen, tool, args, None, latency, thinking,
|
||||
note="repeated a no-op action",
|
||||
parsed_via=parsed.source))
|
||||
episode.stop_reason = (
|
||||
f"stuck: {tool} repeated on an unchanged screen"
|
||||
)
|
||||
return episode
|
||||
recent.append((tool_key, signature))
|
||||
|
||||
result = execute(tool, args, elements, device)
|
||||
if result.ok and not result.done:
|
||||
_, _, after = observe(device)
|
||||
if after == signature:
|
||||
# Say so out loud: the model has no other way to learn that
|
||||
# its action changed nothing.
|
||||
result = ActionResult(True, result.message + " -- the screen did not change",
|
||||
action=result.action)
|
||||
elif tool_key == last_action and after == signature_before_last:
|
||||
# Oscillation: the *same* action, undoing what it just did --
|
||||
# the measured failure, tapping a switch off and on forever.
|
||||
# It hides from the no-change check because each individual tap
|
||||
# really does change the screen. Deliberately narrow: revisiting
|
||||
# a screen by other means (open a submenu, press back) is normal
|
||||
# navigation and must not trip this.
|
||||
result = ActionResult(
|
||||
True,
|
||||
result.message + " -- that undid your previous action; the "
|
||||
"setting is now back where it started",
|
||||
action=result.action,
|
||||
)
|
||||
oscillations += 1
|
||||
if oscillations >= 2:
|
||||
episode.steps.append(Step(n, screen, tool, args, result, latency,
|
||||
thinking, parsed_via=parsed.source))
|
||||
episode.stop_reason = "stuck: repeatedly undoing its own action"
|
||||
return episode
|
||||
signature_before_last, last_action = signature, tool_key
|
||||
episode.steps.append(Step(n, screen, tool, args, result, latency, thinking,
|
||||
parsed_via=parsed.source))
|
||||
log.append(f"{tool} -> {result.message}")
|
||||
|
||||
if result.done:
|
||||
episode.done = True
|
||||
episode.summary = result.summary
|
||||
episode.stop_reason = "task_complete"
|
||||
return episode
|
||||
|
||||
episode.stop_reason = f"step budget exhausted ({cfg.max_steps})"
|
||||
return episode
|
||||
@@ -0,0 +1,354 @@
|
||||
"""Where the agent's actions actually land.
|
||||
|
||||
Two backends behind one interface. `AdbDevice` talks to a phone over `adb`.
|
||||
`SimulatedDevice` runs a small scripted phone in-process, and exists for a
|
||||
reason beyond convenience: no Android device, emulator or `/dev/kvm` is
|
||||
reachable from the host this was built on, so without it the loop, the tool
|
||||
executor and the stopping rules would all be untested code shipped on a
|
||||
promise.
|
||||
|
||||
The simulator emits **real uiautomator XML**, not pre-rendered text, so the
|
||||
parser, the index-to-coordinate mapping and the screen-change detection are
|
||||
exercised on the same path a phone would drive. What it cannot test is
|
||||
whether a real Settings app is laid out the way the fixtures assume; that is
|
||||
stated in the report rather than hidden.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from xml.sax.saxutils import escape as _xml_escape
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable, Protocol
|
||||
|
||||
KEYCODES = {
|
||||
"back": "KEYCODE_BACK",
|
||||
"home": "KEYCODE_HOME",
|
||||
"enter": "KEYCODE_ENTER",
|
||||
"delete": "KEYCODE_DEL",
|
||||
"recent": "KEYCODE_APP_SWITCH",
|
||||
}
|
||||
|
||||
|
||||
class Device(Protocol):
|
||||
def dump(self) -> str: ...
|
||||
def tap(self, x: int, y: int) -> None: ...
|
||||
def input_text(self, text: str) -> None: ...
|
||||
def swipe(self, x1: int, y1: int, x2: int, y2: int, ms: int = 300) -> None: ...
|
||||
def keyevent(self, key: str) -> None: ...
|
||||
def launch(self, app: str) -> None: ...
|
||||
def size(self) -> tuple[int, int]: ...
|
||||
|
||||
|
||||
class AdbError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class AdbDevice:
|
||||
"""A real phone over `adb`."""
|
||||
|
||||
def __init__(self, serial: str | None = None, adb: str = "adb",
|
||||
timeout: float = 30.0):
|
||||
if shutil.which(adb) is None:
|
||||
raise AdbError(
|
||||
f"{adb!r} is not on PATH. Install platform-tools and connect a "
|
||||
"device (`adb devices` should list it)."
|
||||
)
|
||||
self.adb, self.serial, self.timeout = adb, serial, timeout
|
||||
self._size: tuple[int, int] | None = None
|
||||
|
||||
def _run(self, *args: str, binary: bool = False):
|
||||
cmd = [self.adb] + (["-s", self.serial] if self.serial else []) + list(args)
|
||||
proc = subprocess.run(cmd, capture_output=True, timeout=self.timeout)
|
||||
if proc.returncode != 0:
|
||||
raise AdbError(f"{' '.join(cmd)} failed: {proc.stderr.decode(errors='replace')[:400]}")
|
||||
return proc.stdout if binary else proc.stdout.decode(errors="replace")
|
||||
|
||||
def dump(self) -> str:
|
||||
# exec-out keeps the XML on stdout instead of round-tripping through
|
||||
# /sdcard, which is both faster and avoids a writable-storage
|
||||
# assumption that does not hold on every device.
|
||||
out = self._run("exec-out", "uiautomator", "dump", "/dev/tty")
|
||||
start = out.find("<?xml")
|
||||
if start < 0:
|
||||
start = out.find("<hierarchy")
|
||||
if start < 0:
|
||||
raise AdbError(f"no UI hierarchy in dump output: {out[:200]!r}")
|
||||
end = out.rfind("</hierarchy>")
|
||||
return out[start:end + len("</hierarchy>")] if end > 0 else out[start:]
|
||||
|
||||
def tap(self, x: int, y: int) -> None:
|
||||
self._run("shell", "input", "tap", str(x), str(y))
|
||||
|
||||
def input_text(self, text: str) -> None:
|
||||
# `input text` takes a single token: spaces become %s and the shell
|
||||
# metacharacters have to go, or half the string is eaten.
|
||||
escaped = text.replace(" ", "%s")
|
||||
for ch in "()<>|;&*\\~\"'`$":
|
||||
escaped = escaped.replace(ch, "\\" + ch)
|
||||
self._run("shell", "input", "text", escaped)
|
||||
|
||||
def swipe(self, x1: int, y1: int, x2: int, y2: int, ms: int = 300) -> None:
|
||||
self._run("shell", "input", "swipe", str(x1), str(y1), str(x2), str(y2), str(ms))
|
||||
|
||||
def keyevent(self, key: str) -> None:
|
||||
code = KEYCODES.get(key, key)
|
||||
self._run("shell", "input", "keyevent", code)
|
||||
|
||||
def launch(self, app: str) -> None:
|
||||
"""Launch by package name, or by visible label via the launcher."""
|
||||
if "." in app:
|
||||
self._run("shell", "monkey", "-p", app, "-c",
|
||||
"android.intent.category.LAUNCHER", "1")
|
||||
return
|
||||
raise AdbError(
|
||||
f"open_app needs a package name, got {app!r}. Resolve the label to a "
|
||||
"package first (`adb shell pm list packages`) or tap its icon."
|
||||
)
|
||||
|
||||
def size(self) -> tuple[int, int]:
|
||||
if self._size is None:
|
||||
out = self._run("shell", "wm", "size")
|
||||
part = out.strip().rsplit(":", 1)[-1].strip()
|
||||
w, h = part.split("x")
|
||||
self._size = (int(w), int(h))
|
||||
return self._size
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Simulated phone
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class SimElement:
|
||||
label: str
|
||||
cls: str = "android.widget.TextView"
|
||||
clickable: bool = True
|
||||
checkable: bool = False
|
||||
checked: bool = False
|
||||
selected: bool = False
|
||||
editable: bool = False
|
||||
goto: str | None = None
|
||||
"""Screen this element navigates to when tapped."""
|
||||
toggles: str | None = None
|
||||
"""State key this element flips when tapped."""
|
||||
saves: str | None = None
|
||||
"""Record key. Tapping commits the current field values into
|
||||
``records[saves]`` and clears them -- the difference between typing a time
|
||||
and having an alarm, which the scored suite previously could not tell."""
|
||||
label_of: str | None = None
|
||||
"""State key whose value picks the label from ``labels``."""
|
||||
labels: tuple[str, str] = ("", "")
|
||||
"""(label when the state key is false, label when true)."""
|
||||
|
||||
def rendered_label(self, state: dict) -> str:
|
||||
if self.label_of:
|
||||
return self.labels[bool(state.get(self.label_of))]
|
||||
return self.label
|
||||
|
||||
|
||||
@dataclass
|
||||
class SimScreen:
|
||||
screen_id: str
|
||||
elements: list[SimElement]
|
||||
back: str | None = None
|
||||
dynamic: "Callable[[SimulatedDevice], list[SimElement]] | None" = None
|
||||
"""Rows derived from device state, appended after ``elements``. A saved
|
||||
alarm has to appear *on the screen*, or an oracle that reads the screen --
|
||||
which is the only kind a real phone can have -- cannot tell a committed
|
||||
alarm from a typed one."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class SimulatedDevice:
|
||||
"""A scripted phone that speaks uiautomator XML.
|
||||
|
||||
Deliberately small. It is here to prove the loop terminates, recovers and
|
||||
stops -- not to model Android.
|
||||
"""
|
||||
|
||||
screens: dict[str, SimScreen]
|
||||
current: str
|
||||
state: dict[str, bool] = field(default_factory=dict)
|
||||
width: int = 1080
|
||||
height: int = 2400
|
||||
typed: list[str] = field(default_factory=list)
|
||||
actions: list[tuple[str, ...]] = field(default_factory=list)
|
||||
home: str = "home"
|
||||
focus: str | None = None
|
||||
"""Label of the focused text field; typing lands here."""
|
||||
fields: dict[str, str] = field(default_factory=dict)
|
||||
records: dict[str, list[dict]] = field(default_factory=dict)
|
||||
toggled: list[tuple[str, bool, bool]] = field(default_factory=list)
|
||||
"""(key, before, after) per switch tap. A task whose starting state is
|
||||
already correct is failed by a round trip, which final state cannot see."""
|
||||
|
||||
_ROW_H = 160
|
||||
_TOP = 200
|
||||
|
||||
def _visible(self) -> list[SimElement]:
|
||||
screen = self.screens[self.current]
|
||||
rows = list(screen.elements)
|
||||
if screen.dynamic:
|
||||
rows = [r for r in rows if not getattr(r, "_placeholder", False)]
|
||||
rows.extend(screen.dynamic(self))
|
||||
return rows
|
||||
|
||||
def _bounds(self, row: int) -> tuple[int, int, int, int]:
|
||||
y1 = self._TOP + row * self._ROW_H
|
||||
return 0, y1, self.width, y1 + self._ROW_H
|
||||
|
||||
def dump(self) -> str:
|
||||
rows = []
|
||||
for i, el in enumerate(self._visible()):
|
||||
x1, y1, x2, y2 = self._bounds(i)
|
||||
checked = el.checked
|
||||
if el.toggles:
|
||||
checked = self.state.get(el.toggles, el.checked)
|
||||
# uiautomator escapes attribute values, and labels really do
|
||||
# contain them -- "Network & internet" is on every Android
|
||||
# settings screen. Emitting raw breaks the parser on the second
|
||||
# screen of the first task.
|
||||
label = _xml_escape(el.rendered_label(self.state), {'"': """})
|
||||
rows.append(
|
||||
f' <node index="{i}" text="{label}" resource-id="" '
|
||||
f'class="{el.cls}" package="com.example.sim" content-desc="" '
|
||||
f'checkable="{str(el.checkable).lower()}" '
|
||||
f'checked="{str(checked).lower()}" '
|
||||
f'clickable="{str(el.clickable).lower()}" enabled="true" '
|
||||
f'focusable="{str(el.clickable).lower()}" focused="false" '
|
||||
f'scrollable="false" long-clickable="false" password="false" '
|
||||
f'selected="{str(el.selected).lower()}" '
|
||||
f'bounds="[{x1},{y1}][{x2},{y2}]" />'
|
||||
)
|
||||
body = "\n".join(rows)
|
||||
return (
|
||||
'<?xml version="1.0" encoding="UTF-8"?>\n'
|
||||
f'<hierarchy rotation="0">\n{body}\n</hierarchy>'
|
||||
)
|
||||
|
||||
def _row_at(self, y: int) -> int | None:
|
||||
row = (y - self._TOP) // self._ROW_H
|
||||
return row if 0 <= row < len(self._visible()) else None
|
||||
|
||||
def tap(self, x: int, y: int) -> None:
|
||||
self.actions.append(("tap", str(x), str(y)))
|
||||
row = self._row_at(y)
|
||||
if row is None:
|
||||
return
|
||||
el = self._visible()[row]
|
||||
if el.editable:
|
||||
self.focus = el.label
|
||||
if el.toggles:
|
||||
before = bool(self.state.get(el.toggles, el.checked))
|
||||
self.state[el.toggles] = not before
|
||||
self.toggled.append((el.toggles, before, not before))
|
||||
if el.saves:
|
||||
self.records.setdefault(el.saves, []).append(
|
||||
{**self.fields, **{k: v for k, v in self.state.items()
|
||||
if k.startswith(el.saves + "_")}})
|
||||
self.fields = {}
|
||||
self.focus = None
|
||||
if el.goto and el.goto in self.screens:
|
||||
self.current = el.goto
|
||||
|
||||
def input_text(self, text: str) -> None:
|
||||
self.actions.append(("text", text))
|
||||
self.typed.append(text)
|
||||
if self.focus is not None:
|
||||
self.fields[self.focus] = self.fields.get(self.focus, "") + text
|
||||
|
||||
def swipe(self, x1: int, y1: int, x2: int, y2: int, ms: int = 300) -> None:
|
||||
self.actions.append(("swipe", str(x1), str(y1), str(x2), str(y2)))
|
||||
|
||||
def keyevent(self, key: str) -> None:
|
||||
self.actions.append(("key", key))
|
||||
if key == "back":
|
||||
self.current = self.screens[self.current].back or self.home
|
||||
elif key == "home":
|
||||
self.current = self.home
|
||||
|
||||
def launch(self, app: str) -> None:
|
||||
self.actions.append(("launch", app))
|
||||
for screen in self.screens.values():
|
||||
for el in screen.elements:
|
||||
if el.label.lower() == app.lower() and el.goto:
|
||||
self.current = el.goto
|
||||
return
|
||||
|
||||
def size(self) -> tuple[int, int]:
|
||||
return self.width, self.height
|
||||
|
||||
|
||||
def _saved_alarm_rows(device: "SimulatedDevice") -> list[SimElement]:
|
||||
"""Render committed alarms the way a clock app lists them."""
|
||||
saved = device.records.get("alarm", [])
|
||||
if not saved:
|
||||
return [SimElement("No alarms", clickable=False)]
|
||||
rows = []
|
||||
for rec in saved:
|
||||
hour = (rec.get("Hour") or "?").lstrip("0") or "0"
|
||||
minute = (rec.get("Minute") or "?").rjust(2, "0")
|
||||
meridiem = "PM" if rec.get("alarm_pm") else "AM"
|
||||
rows.append(SimElement(f"{hour}:{minute} {meridiem}", clickable=False))
|
||||
return rows
|
||||
|
||||
|
||||
def demo_phone() -> SimulatedDevice:
|
||||
"""A two-app phone: Settings (Wi-Fi) and Clock (alarms).
|
||||
|
||||
Enough to exercise a multi-screen task, a stateful toggle -- the case the
|
||||
earlier single-step run got wrong, tapping a switch that was already on --
|
||||
and a text field.
|
||||
"""
|
||||
screens = {
|
||||
"home": SimScreen("home", [
|
||||
SimElement("Phone"),
|
||||
SimElement("Messages"),
|
||||
SimElement("Clock", goto="clock"),
|
||||
SimElement("Settings", goto="settings"),
|
||||
SimElement("Camera"),
|
||||
]),
|
||||
"settings": SimScreen("settings", [
|
||||
SimElement("Settings", clickable=False),
|
||||
SimElement("Network & internet", goto="network"),
|
||||
SimElement("Connected devices"),
|
||||
SimElement("Apps"),
|
||||
SimElement("Battery"),
|
||||
], back="home"),
|
||||
"network": SimScreen("network", [
|
||||
SimElement("Network & internet", clickable=False),
|
||||
SimElement("Wi-Fi", cls="android.widget.Switch", checkable=True,
|
||||
toggles="wifi"),
|
||||
SimElement("Mobile network"),
|
||||
SimElement("Airplane mode", cls="android.widget.Switch",
|
||||
checkable=True, toggles="airplane"),
|
||||
], back="settings"),
|
||||
"clock": SimScreen("clock", [
|
||||
SimElement("Alarm", cls="android.widget.Button", goto="alarms"),
|
||||
SimElement("Clock", cls="android.widget.Button", selected=True),
|
||||
SimElement("Timer", cls="android.widget.Button"),
|
||||
SimElement("Stopwatch", cls="android.widget.Button"),
|
||||
], back="home"),
|
||||
"alarms": SimScreen("alarms", [
|
||||
SimElement("Alarm", cls="android.widget.Button", selected=True),
|
||||
SimElement("Add alarm", cls="android.widget.ImageButton",
|
||||
goto="new_alarm"),
|
||||
], back="clock", dynamic=_saved_alarm_rows),
|
||||
"new_alarm": SimScreen("new_alarm", [
|
||||
SimElement("Set time", clickable=False),
|
||||
SimElement("Hour", cls="android.widget.EditText", editable=True),
|
||||
SimElement("Minute", cls="android.widget.EditText", editable=True),
|
||||
# Android shows one AM/PM control whose label is its value, so
|
||||
# "set an alarm for 7:30 AM" has a wrong answer as well as a right
|
||||
# one. Without it the task could be passed by typing any digits.
|
||||
SimElement("AM", cls="android.widget.Button",
|
||||
label_of="alarm_pm", labels=("AM", "PM"),
|
||||
toggles="alarm_pm"),
|
||||
SimElement("Save", cls="android.widget.Button", goto="alarms",
|
||||
saves="alarm"),
|
||||
], back="alarms"),
|
||||
}
|
||||
return SimulatedDevice(screens=screens, current="home", state={"wifi": False})
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Turn one tool call into one device action.
|
||||
|
||||
Every failure here is returned to the model as a sentence it can act on, not
|
||||
raised. A 1.7B model asked for `index: 9` on a four-element screen has made a
|
||||
recoverable mistake, and "index 9 is out of range, the screen has 4 elements
|
||||
(0-3)" is the difference between the next step being right and the episode
|
||||
ending. Exceptions are reserved for the device being gone.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from device import KEYCODES, Device
|
||||
from ui_tree import Element
|
||||
|
||||
|
||||
@dataclass
|
||||
class ActionResult:
|
||||
ok: bool
|
||||
message: str
|
||||
done: bool = False
|
||||
summary: str = ""
|
||||
action: str = ""
|
||||
|
||||
def as_tool_reply(self) -> str:
|
||||
return self.message
|
||||
|
||||
|
||||
def _need(args: dict, key: str, kind: type, what: str) -> tuple[object, str | None]:
|
||||
if key not in args:
|
||||
return None, f"missing required argument {key!r} for {what}"
|
||||
value = args[key]
|
||||
if kind is int:
|
||||
try:
|
||||
value = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None, f"{key!r} must be a whole number, got {args[key]!r}"
|
||||
elif not isinstance(value, kind):
|
||||
return None, f"{key!r} must be {kind.__name__}, got {type(value).__name__}"
|
||||
return value, None
|
||||
|
||||
|
||||
def _resolve(index: object, elements: list[Element]) -> tuple[Element | None, str | None]:
|
||||
i = int(index)
|
||||
if not elements:
|
||||
return None, "the screen has no addressable elements; try press_key(back)"
|
||||
if not 0 <= i < len(elements):
|
||||
return None, (
|
||||
f"index {i} is out of range; the screen has {len(elements)} "
|
||||
f"elements (0-{len(elements) - 1})"
|
||||
)
|
||||
return elements[i], None
|
||||
|
||||
|
||||
def execute(name: str, args: dict, elements: list[Element],
|
||||
device: Device) -> ActionResult:
|
||||
"""Run one tool call. Returns what to tell the model."""
|
||||
args = args or {}
|
||||
|
||||
if name == "task_complete":
|
||||
summary = str(args.get("summary", "")).strip()
|
||||
return ActionResult(True, "task marked complete", done=True,
|
||||
summary=summary, action="task_complete")
|
||||
|
||||
if name == "tap":
|
||||
index, err = _need(args, "index", int, "tap")
|
||||
if err:
|
||||
return ActionResult(False, err, action="tap")
|
||||
el, err = _resolve(index, elements)
|
||||
if err:
|
||||
return ActionResult(False, err, action="tap")
|
||||
x, y = el.center
|
||||
device.tap(x, y)
|
||||
state = ""
|
||||
if "checked" in el.flags:
|
||||
state = f" (was {'on' if el.flags['checked'] else 'off'})"
|
||||
return ActionResult(True, f'tapped [{index}] "{el.text}"{state}',
|
||||
action=f"tap:{index}")
|
||||
|
||||
if name == "type_text":
|
||||
index, err = _need(args, "index", int, "type_text")
|
||||
if err:
|
||||
return ActionResult(False, err, action="type_text")
|
||||
text, err = _need(args, "text", str, "type_text")
|
||||
if err:
|
||||
return ActionResult(False, err, action="type_text")
|
||||
el, err = _resolve(index, elements)
|
||||
if err:
|
||||
return ActionResult(False, err, action="type_text")
|
||||
# Focus first: `input text` goes to whatever holds focus, which on a
|
||||
# freshly dumped screen is usually nothing.
|
||||
x, y = el.center
|
||||
device.tap(x, y)
|
||||
device.input_text(str(text))
|
||||
return ActionResult(True, f'typed "{text}" into [{index}] "{el.text}"',
|
||||
action=f"type:{index}")
|
||||
|
||||
if name == "swipe":
|
||||
direction = str(args.get("direction", "")).lower()
|
||||
if direction not in ("up", "down", "left", "right"):
|
||||
return ActionResult(
|
||||
False, "direction must be one of up, down, left, right",
|
||||
action="swipe")
|
||||
w, h = device.size()
|
||||
cx, cy = w // 2, h // 2
|
||||
dx, dy = int(w * 0.35), int(h * 0.30)
|
||||
# `direction` is where the *content* should move, i.e. what the user
|
||||
# means by "scroll down to see more", so the finger travels the other
|
||||
# way. Stated in the tool description too, because guessing this wrong
|
||||
# is invisible in the transcript.
|
||||
moves = {
|
||||
"down": (cx, cy + dy, cx, cy - dy),
|
||||
"up": (cx, cy - dy, cx, cy + dy),
|
||||
"right": (cx + dx, cy, cx - dx, cy),
|
||||
"left": (cx - dx, cy, cx + dx, cy),
|
||||
}
|
||||
device.swipe(*moves[direction])
|
||||
return ActionResult(True, f"swiped to scroll {direction}",
|
||||
action=f"swipe:{direction}")
|
||||
|
||||
if name == "press_key":
|
||||
key = str(args.get("key", "")).lower()
|
||||
if key not in KEYCODES:
|
||||
return ActionResult(
|
||||
False, f"unknown key {key!r}; use one of {', '.join(KEYCODES)}",
|
||||
action="press_key")
|
||||
device.keyevent(key)
|
||||
return ActionResult(True, f"pressed {key}", action=f"key:{key}")
|
||||
|
||||
if name == "open_app":
|
||||
app = str(args.get("name", "")).strip()
|
||||
if not app:
|
||||
return ActionResult(False, "open_app needs a name", action="open_app")
|
||||
try:
|
||||
device.launch(app)
|
||||
except Exception as exc: # device-specific launch failures are recoverable
|
||||
return ActionResult(False, f"could not launch {app!r}: {exc}",
|
||||
action="open_app")
|
||||
return ActionResult(True, f"launched {app}", action=f"open:{app}")
|
||||
|
||||
return ActionResult(
|
||||
False,
|
||||
f"unknown tool {name!r}; available: tap, type_text, swipe, press_key, "
|
||||
"open_app, task_complete",
|
||||
action="unknown",
|
||||
)
|
||||
@@ -0,0 +1,135 @@
|
||||
"""What the phone is actually in, read the same way for both backends.
|
||||
|
||||
The scored suite used to take its predicates over ``SimulatedDevice``'s Python
|
||||
attributes, so every scored run was a simulator run by construction -- the
|
||||
real-device path existed for *driving* a phone and not for judging one. That
|
||||
made "the agent suite" a claim about fixtures.
|
||||
|
||||
Success is therefore read two ways, and the order matters:
|
||||
|
||||
1. **Out of band, from the setting itself** -- ``device.state`` in the
|
||||
simulator, ``settings get global wifi_on`` over adb. This is the ground
|
||||
truth, and it does not depend on where the agent happened to leave the
|
||||
phone. Scoring from the final screen alone fails an agent that turns Wi-Fi
|
||||
on correctly and then presses Home, which is not a failure.
|
||||
2. **From the UI dump**, for anything with no setting behind it -- a saved
|
||||
alarm appears as a row in the clock app and nowhere else. The simulator
|
||||
emits real uiautomator XML, so this goes down the same parse, the same
|
||||
switch lookup and the same text matching a phone would.
|
||||
|
||||
Only the facts a phone genuinely cannot report after the fact -- which
|
||||
switches were flipped on the way, and what was typed into which field -- come
|
||||
from the simulator object, and any task needing them declares it, so a
|
||||
real-device run refuses to score rather than scoring wrong.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from ui_tree import parse_ui_dump
|
||||
|
||||
|
||||
def _norm(s: str) -> str:
|
||||
"""Match labels across the punctuation Android is inconsistent about."""
|
||||
return "".join(c for c in s.lower() if c.isalnum())
|
||||
|
||||
|
||||
@dataclass
|
||||
class PhoneState:
|
||||
source: str
|
||||
"""``"sim"`` or ``"device"`` -- which facts below are trustworthy."""
|
||||
settings: dict[str, bool] = field(default_factory=dict)
|
||||
"""Authoritative setting values, read out of band. Preferred over
|
||||
:attr:`switches`, which only shows what is currently on screen."""
|
||||
screen: str | None = None
|
||||
"""Simulator screen id. ``None`` on a real phone, where there is no such
|
||||
thing; use :meth:`has_text` against something the screen actually shows."""
|
||||
switches: dict[str, bool] = field(default_factory=dict)
|
||||
texts: list[str] = field(default_factory=list)
|
||||
typed: list[str] = field(default_factory=list)
|
||||
fields: dict[str, str] = field(default_factory=dict)
|
||||
records: dict[str, list[dict]] = field(default_factory=dict)
|
||||
toggled: list[tuple[str, bool, bool]] = field(default_factory=list)
|
||||
"""(key, before, after) for every switch the agent actually flipped."""
|
||||
|
||||
def switch(self, *names: str) -> bool | None:
|
||||
"""State of the first control matching any of ``names``.
|
||||
|
||||
The setting wins over the screen: an agent that turned Wi-Fi on and
|
||||
then navigated away has still turned Wi-Fi on. ``None`` means neither
|
||||
source knows the control, which is not the same as off -- an agent that
|
||||
never found the screen has not turned anything on, and predicates must
|
||||
not read that as "off".
|
||||
"""
|
||||
wanted = {_norm(n) for n in names}
|
||||
for key, value in self.settings.items():
|
||||
if _norm(key) in wanted:
|
||||
return bool(value)
|
||||
for label, checked in self.switches.items():
|
||||
if _norm(label) in wanted:
|
||||
return checked
|
||||
return None
|
||||
|
||||
def has_text(self, *needles: str) -> bool:
|
||||
joined = " ".join(self.texts).lower()
|
||||
return any(n.lower() in joined for n in needles)
|
||||
|
||||
def times_toggled(self, key: str) -> int:
|
||||
return sum(1 for k, _, _ in self.toggled if k == key)
|
||||
|
||||
|
||||
def read_state(device) -> PhoneState:
|
||||
"""Read the phone through its screen, plus the simulator's own records."""
|
||||
elements = parse_ui_dump(device.dump())
|
||||
# `checkable` is deliberately not among the flags ui_tree parses, because
|
||||
# that list is also what the model is shown; adding to it would change the
|
||||
# agent's prompt. A switch is recognised by its class or by exposing a
|
||||
# `checked` flag instead.
|
||||
switches = {e.text: bool(e.flags.get("checked", False))
|
||||
for e in elements
|
||||
if "checked" in e.flags
|
||||
or e.short_class in ("Switch", "CheckBox", "ToggleButton")}
|
||||
texts = [e.text for e in elements if e.text]
|
||||
|
||||
if not hasattr(device, "screens"):
|
||||
return PhoneState(source="device", switches=switches, texts=texts,
|
||||
settings=adb_settings(device))
|
||||
|
||||
return PhoneState(
|
||||
source="sim",
|
||||
screen=device.current,
|
||||
settings={k: bool(v) for k, v in device.state.items()},
|
||||
switches=switches,
|
||||
texts=texts,
|
||||
typed=list(device.typed),
|
||||
fields=dict(getattr(device, "fields", {})),
|
||||
records={k: [dict(r) for r in v]
|
||||
for k, v in getattr(device, "records", {}).items()},
|
||||
toggled=list(getattr(device, "toggled", [])),
|
||||
)
|
||||
|
||||
|
||||
ADB_SETTINGS = {
|
||||
"wifi": ("global", "wifi_on"),
|
||||
"airplane": ("global", "airplane_mode_on"),
|
||||
"bluetooth": ("global", "bluetooth_on"),
|
||||
}
|
||||
"""Settings readable out of band, so scoring does not depend on which screen
|
||||
the agent left the phone on."""
|
||||
|
||||
|
||||
def adb_settings(device) -> dict[str, bool]:
|
||||
"""Read the authoritative values from a real phone. Missing keys are
|
||||
omitted rather than defaulted -- an unreadable setting must not be scored
|
||||
as "off"."""
|
||||
out: dict[str, bool] = {}
|
||||
for key, (namespace, name) in ADB_SETTINGS.items():
|
||||
try:
|
||||
raw = device._run("shell", "settings", "get", namespace, name)
|
||||
except Exception:
|
||||
continue
|
||||
raw = raw.strip()
|
||||
if raw in ("0", "1"):
|
||||
out[key] = raw == "1"
|
||||
return out
|
||||
@@ -0,0 +1,98 @@
|
||||
"""Drive Spark-X2.5 through Android control steps against an OpenAI endpoint.
|
||||
|
||||
Measures whether the model picks the right tool and the right element, and how
|
||||
long a decision takes -- the two things that decide whether a 1.7B model can
|
||||
actually run a phone agent locally.
|
||||
"""
|
||||
import argparse, json, time
|
||||
from urllib import request as urlrequest
|
||||
|
||||
from action_space import SYSTEM_PROMPT, TOOLS
|
||||
from scenarios import SCENARIOS
|
||||
|
||||
|
||||
def chat(base_url, model, messages, tools, timeout=900, max_tokens=1024,
|
||||
think=True):
|
||||
payload = {
|
||||
"model": model, "messages": messages, "tools": tools,
|
||||
"tool_choice": "auto", "temperature": 0.0, "max_tokens": max_tokens,
|
||||
}
|
||||
if not think:
|
||||
# Spark's template opens <think> by default; closing it up front makes
|
||||
# the model answer directly, which is what a phone agent wants.
|
||||
payload["chat_template_kwargs"] = {"enable_thinking": False}
|
||||
body = json.dumps(payload).encode()
|
||||
req = urlrequest.Request(f"{base_url}/chat/completions", data=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
t0 = time.perf_counter()
|
||||
with urlrequest.urlopen(req, timeout=timeout) as r:
|
||||
out = json.loads(r.read())
|
||||
return out, time.perf_counter() - t0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--base-url", default="http://127.0.0.1:8077/v1")
|
||||
ap.add_argument("--model", default="XHToken/Spark-X2.5-1.7B")
|
||||
ap.add_argument("--no-think", action="store_true")
|
||||
ap.add_argument("--max-tokens", type=int, default=1024)
|
||||
ap.add_argument("--out", default="android_agent_results.json")
|
||||
a = ap.parse_args()
|
||||
|
||||
rows, correct_tool, correct_args = [], 0, 0
|
||||
for sc in SCENARIOS:
|
||||
msgs = [
|
||||
{"role": "system", "content": SYSTEM_PROMPT},
|
||||
{"role": "user", "content":
|
||||
f"Task: {sc['task']}\n\nCurrent screen:\n{sc['screen']}\n\n"
|
||||
"Choose the next action."},
|
||||
]
|
||||
try:
|
||||
out, dt = chat(a.base_url, a.model, msgs, TOOLS,
|
||||
max_tokens=a.max_tokens, think=not a.no_think)
|
||||
except Exception as e:
|
||||
rows.append({"id": sc["id"], "error": str(e)[:200]})
|
||||
print(f"{sc['id']:<20} ERROR {str(e)[:90]}", flush=True)
|
||||
continue
|
||||
msg = out["choices"][0]["message"]
|
||||
calls = msg.get("tool_calls") or []
|
||||
got_name = calls[0]["function"]["name"] if calls else None
|
||||
try:
|
||||
got_args = json.loads(calls[0]["function"]["arguments"]) if calls else {}
|
||||
except Exception:
|
||||
got_args = {"_raw": calls[0]["function"]["arguments"]}
|
||||
|
||||
name_ok = got_name in sc["accept_names"]
|
||||
args_ok = name_ok and all(
|
||||
str(got_args.get(k)) == str(v) for k, v in sc["expect"]["args"].items()
|
||||
) if got_name == sc["expect"]["name"] else name_ok and not sc["expect"]["args"]
|
||||
correct_tool += int(name_ok)
|
||||
correct_args += int(bool(args_ok))
|
||||
rows.append({
|
||||
"id": sc["id"], "latency_s": round(dt, 2),
|
||||
"expected": sc["expect"], "got_name": got_name, "got_args": got_args,
|
||||
"tool_ok": name_ok, "args_ok": bool(args_ok),
|
||||
"reasoning_chars": len(msg.get("reasoning_content") or ""),
|
||||
"completion_tokens": out.get("usage", {}).get("completion_tokens"),
|
||||
})
|
||||
flag = "OK " if args_ok else ("~ " if name_ok else "X ")
|
||||
print(f"{flag}{sc['id']:<20} {got_name}({got_args}) {dt:.1f}s", flush=True)
|
||||
|
||||
n = len(SCENARIOS)
|
||||
summary = {
|
||||
"scenarios": n,
|
||||
"thinking": not a.no_think,
|
||||
"tool_accuracy": f"{correct_tool}/{n}",
|
||||
"action_accuracy": f"{correct_args}/{n}",
|
||||
"median_latency_s": round(
|
||||
sorted(r["latency_s"] for r in rows if "latency_s" in r)[
|
||||
max(0, len([r for r in rows if "latency_s" in r]) // 2)] , 2)
|
||||
if any("latency_s" in r for r in rows) else None,
|
||||
"rows": rows,
|
||||
}
|
||||
json.dump(summary, open(a.out, "w"), indent=2)
|
||||
print(json.dumps({k: v for k, v in summary.items() if k != "rows"}, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Drive Spark-X2.5 through a whole phone task, not one decision.
|
||||
|
||||
# against the in-process simulated phone (no device needed)
|
||||
python run_episode.py --task "Turn on Wi-Fi." --device sim
|
||||
|
||||
# against a real phone
|
||||
adb devices # confirm it is listed
|
||||
python run_episode.py --task "Turn on Wi-Fi." --device adb
|
||||
|
||||
The server side is vLLM's stock OpenAI endpoint; Spark emits GLM-4.5's tool
|
||||
format, so no custom parser is needed:
|
||||
|
||||
vllm serve <model> --port 8077 --enable-auto-tool-choice \\
|
||||
--tool-call-parser glm45 --reasoning-parser glm45
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
|
||||
from agent import AgentConfig, run_episode
|
||||
from device import AdbDevice, AdbError, demo_phone
|
||||
|
||||
TASKS = {
|
||||
"wifi": "Turn on Wi-Fi.",
|
||||
"alarm": "Set an alarm for 7:30 AM.",
|
||||
}
|
||||
|
||||
|
||||
def build_device(kind: str, serial: str | None):
|
||||
if kind == "sim":
|
||||
return demo_phone(), "simulated phone (settings + clock)"
|
||||
try:
|
||||
device = AdbDevice(serial=serial)
|
||||
w, h = device.size()
|
||||
return device, f"adb device {serial or '(default)'} {w}x{h}"
|
||||
except AdbError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--task", default=TASKS["wifi"],
|
||||
help=f"free text, or one of: {', '.join(TASKS)}")
|
||||
ap.add_argument("--device", choices=("sim", "adb"), default="sim")
|
||||
ap.add_argument("--serial", default=None)
|
||||
ap.add_argument("--base-url", default="http://127.0.0.1:8077/v1")
|
||||
ap.add_argument("--model", default="XHToken/Spark-X2.5-1.7B")
|
||||
ap.add_argument("--max-steps", type=int, default=12)
|
||||
ap.add_argument("--max-tokens", type=int, default=1024)
|
||||
ap.add_argument("--no-think", action="store_true")
|
||||
ap.add_argument("--no-think-fallback", action="store_true")
|
||||
ap.add_argument("--out", default=None)
|
||||
a = ap.parse_args()
|
||||
|
||||
task = TASKS.get(a.task, a.task)
|
||||
device, described = build_device(a.device, a.serial)
|
||||
cfg = AgentConfig(
|
||||
base_url=a.base_url, model=a.model, max_steps=a.max_steps,
|
||||
max_tokens=a.max_tokens, think=not a.no_think,
|
||||
think_fallback=not a.no_think_fallback,
|
||||
)
|
||||
|
||||
print(f"device: {described}")
|
||||
print(f"task: {task}")
|
||||
t0 = time.perf_counter()
|
||||
episode = run_episode(device, task, cfg)
|
||||
wall = time.perf_counter() - t0
|
||||
|
||||
print()
|
||||
print(episode.transcript())
|
||||
latencies = [s.latency_s for s in episode.steps]
|
||||
print(f"\n{episode.n_steps} steps in {wall:.1f} s"
|
||||
f" (median decision {sorted(latencies)[len(latencies) // 2]:.2f} s)"
|
||||
if latencies else f"\n{episode.n_steps} steps in {wall:.1f} s")
|
||||
if hasattr(device, "state"):
|
||||
print(f"device state: {device.state}")
|
||||
|
||||
if a.out:
|
||||
payload = {
|
||||
"task": task, "device": described, "done": episode.done,
|
||||
"summary": episode.summary, "stop_reason": episode.stop_reason,
|
||||
"wall_s": round(wall, 2),
|
||||
"think": cfg.think, "max_steps": cfg.max_steps,
|
||||
"steps": [
|
||||
{"n": s.n, "tool": s.tool, "args": s.args,
|
||||
"ok": s.result.ok if s.result else False,
|
||||
"message": s.result.message if s.result else s.note,
|
||||
"latency_s": round(s.latency_s, 3), "thinking": s.thinking}
|
||||
for s in episode.steps
|
||||
],
|
||||
"final_state": getattr(device, "state", None),
|
||||
}
|
||||
with open(a.out, "w") as f:
|
||||
json.dump(payload, f, indent=2, ensure_ascii=False)
|
||||
print(f"wrote {a.out}")
|
||||
|
||||
raise SystemExit(0 if episode.done else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Run the task suite and score it, so an agent change can be compared.
|
||||
|
||||
Success is decided by the phone's own settings and screen, not by the
|
||||
transcript: an agent that says it turned Wi-Fi on and left it off fails here,
|
||||
which is exactly the failure this model exhibits.
|
||||
|
||||
Two things changed after the plan review:
|
||||
|
||||
* ``--repeat N`` now walks the task's **variants** rather than re-running one
|
||||
fixture. At temperature 0 against fixed fixtures, repeats measure
|
||||
repeatability; they only become trials if the phone differs between them.
|
||||
The summary reports a pass rate per task with the variants named, so a build
|
||||
comparison rests on more than one cell.
|
||||
* ``--device adb`` scores a real phone. Each run resets the device to the
|
||||
task's declared starting state first, and a task that cannot be scored from
|
||||
a phone (because its predicate needs flip history) is skipped and reported
|
||||
as skipped, rather than silently scored from final state.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import statistics
|
||||
import time
|
||||
from collections import defaultdict
|
||||
|
||||
from agent import AgentConfig, run_episode, wait_for_server
|
||||
from phone_state import read_state
|
||||
from tasks import SUITE, SUITE_BY_ID
|
||||
|
||||
|
||||
def _reset_real_device(device, task) -> None:
|
||||
if not task.reset_adb:
|
||||
raise SystemExit(
|
||||
f"task {task.task_id!r} declares no reset_adb, so a phone run would "
|
||||
f"be scored against an unknown starting state. Add one or exclude "
|
||||
f"the task with --tasks."
|
||||
)
|
||||
for cmd in task.reset_adb:
|
||||
device._run(*cmd)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--base-url", default="http://127.0.0.1:8077/v1")
|
||||
ap.add_argument("--model", default="spark")
|
||||
ap.add_argument("--tasks", default="all",
|
||||
help="comma-separated task ids, or 'all'")
|
||||
ap.add_argument("--repeat", type=int, default=1,
|
||||
help="runs per task; each walks the next variant")
|
||||
ap.add_argument("--device", choices=("sim", "adb"), default="sim")
|
||||
ap.add_argument("--serial", default=None, help="adb device serial")
|
||||
ap.add_argument("--no-think", action="store_true")
|
||||
ap.add_argument("--max-tokens", type=int, default=1024)
|
||||
ap.add_argument("--label", default="")
|
||||
ap.add_argument("--out", default=None)
|
||||
a = ap.parse_args()
|
||||
|
||||
tasks = SUITE if a.tasks == "all" else [SUITE_BY_ID[t] for t in a.tasks.split(",")]
|
||||
cfg = AgentConfig(base_url=a.base_url, model=a.model,
|
||||
think=not a.no_think, max_tokens=a.max_tokens)
|
||||
|
||||
phone = None
|
||||
if a.device == "adb":
|
||||
from device import AdbDevice
|
||||
phone = AdbDevice(serial=a.serial)
|
||||
skipped = [t.task_id for t in tasks if t.needs]
|
||||
if skipped:
|
||||
print(f"skipping on a real phone (needs flip history): "
|
||||
f"{', '.join(skipped)}")
|
||||
tasks = [t for t in tasks if not t.needs]
|
||||
|
||||
if not wait_for_server(cfg, timeout_s=600):
|
||||
raise SystemExit(f"no model server at {a.base_url}")
|
||||
|
||||
rows, t0 = [], time.perf_counter()
|
||||
for task in tasks:
|
||||
for run in range(a.repeat):
|
||||
if phone is not None:
|
||||
_reset_real_device(phone, task)
|
||||
device = phone
|
||||
else:
|
||||
device = task.build(run)
|
||||
cfg.max_steps = task.max_steps
|
||||
episode = run_episode(device, task.prompt, cfg)
|
||||
score = task.score(read_state(device), episode.done, run)
|
||||
score.update(
|
||||
run=run + 1, steps=episode.n_steps,
|
||||
stop_reason=episode.stop_reason,
|
||||
recovered_steps=sum(1 for s in episode.steps
|
||||
if s.parsed_via not in ("tool_calls", "none")),
|
||||
median_latency_s=round(
|
||||
sorted(s.latency_s for s in episode.steps)[episode.n_steps // 2], 2)
|
||||
if episode.n_steps else None,
|
||||
)
|
||||
rows.append(score)
|
||||
mark = "PASS" if score["success"] else "fail"
|
||||
detail = []
|
||||
if not score.get("scorable", True):
|
||||
mark, detail = "skip", [score.get("reason", "")]
|
||||
else:
|
||||
if score["reached_goal"] and not score["declared_done"]:
|
||||
detail.append("goal reached but never stopped")
|
||||
if score["declared_done"] and not score["reached_goal"]:
|
||||
detail.append("claimed done, goal NOT reached")
|
||||
if score["collateral_damage"]:
|
||||
detail.append("collateral damage")
|
||||
print(f" {mark} {task.task_id:18s} {score['variant']:10s} "
|
||||
f"steps={score['steps']:2d} "
|
||||
f"{score['stop_reason'][:34]:36s} {'; '.join(detail)}", flush=True)
|
||||
|
||||
wall = time.perf_counter() - t0
|
||||
scored = [r for r in rows if r.get("scorable", True)]
|
||||
n_pass = sum(1 for r in scored if r["success"])
|
||||
reached = sum(1 for r in scored if r["reached_goal"])
|
||||
claimed = sum(1 for r in scored if r["declared_done"] and not r["reached_goal"])
|
||||
|
||||
by_task: dict[str, list[dict]] = defaultdict(list)
|
||||
for r in scored:
|
||||
by_task[r["task_id"]].append(r)
|
||||
print("\npass rate per task")
|
||||
for task_id, rs in by_task.items():
|
||||
passes = sum(1 for r in rs if r["success"])
|
||||
failed = [r["variant"] for r in rs if not r["success"]]
|
||||
print(f" {task_id:18s} {passes}/{len(rs)}"
|
||||
+ (f" failed on: {', '.join(failed)}" if failed else ""))
|
||||
|
||||
rates = [sum(1 for r in rs if r["success"]) / len(rs) for rs in by_task.values()]
|
||||
spread = (f"{min(rates):.0%}-{max(rates):.0%}" if rates else "-")
|
||||
print(f"\n{n_pass}/{len(scored)} passed in {wall:.0f}s"
|
||||
f" (goal reached {reached}/{len(scored)}; "
|
||||
f"falsely claimed done {claimed}; per-task rate {spread}"
|
||||
+ (f"; median {statistics.median(rates):.0%}" if rates else "") + ")")
|
||||
if len(scored) and len(by_task) and max(len(rs) for rs in by_task.values()) < 3:
|
||||
print(" NOTE: fewer than 3 runs per task -- enough to show a failure "
|
||||
"reproduces, not to rank builds.")
|
||||
|
||||
if a.out:
|
||||
with open(a.out, "w") as f:
|
||||
json.dump({"label": a.label, "base_url": a.base_url,
|
||||
"model": a.model, "think": cfg.think,
|
||||
"device": a.device, "repeat": a.repeat,
|
||||
"passed": n_pass, "total": len(scored),
|
||||
"skipped": len(rows) - len(scored),
|
||||
"wall_s": round(wall, 1), "rows": rows}, f, indent=2)
|
||||
print(f"wrote {a.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,113 @@
|
||||
"""Android screens as a text-only agent sees them, with the correct next action.
|
||||
|
||||
Each screen is a uiautomator-style accessibility dump flattened to one line per
|
||||
element -- the same shape droidrun's Portal and AndroidWorld's a11y observation
|
||||
hand to a text model. `expect` is the action a competent agent must take next;
|
||||
`accept_names` allows a defensible alternative where one genuinely exists.
|
||||
"""
|
||||
|
||||
SCENARIOS = [
|
||||
{
|
||||
"id": "launch_clock",
|
||||
"task": "Set an alarm for 7:30 AM.",
|
||||
"screen": """[0] TextView "Phone" clickable=true
|
||||
[1] TextView "Messages" clickable=true
|
||||
[2] TextView "Chrome" clickable=true
|
||||
[3] TextView "Clock" clickable=true
|
||||
[4] TextView "Settings" clickable=true
|
||||
[5] TextView "Camera" clickable=true""",
|
||||
"expect": {"name": "tap", "args": {"index": 3}},
|
||||
"accept_names": ["tap", "open_app"],
|
||||
},
|
||||
{
|
||||
"id": "clock_alarm_tab",
|
||||
"task": "Set an alarm for 7:30 AM.",
|
||||
"screen": """[0] Button "Alarm" clickable=true selected=false
|
||||
[1] Button "Clock" clickable=true selected=true
|
||||
[2] Button "Timer" clickable=true selected=false
|
||||
[3] Button "Stopwatch" clickable=true selected=false
|
||||
[4] TextView "14:32" clickable=false""",
|
||||
"expect": {"name": "tap", "args": {"index": 0}},
|
||||
"accept_names": ["tap"],
|
||||
},
|
||||
{
|
||||
"id": "add_alarm",
|
||||
"task": "Set an alarm for 7:30 AM.",
|
||||
"screen": """[0] Button "Alarm" clickable=true selected=true
|
||||
[1] TextView "No alarms" clickable=false
|
||||
[2] ImageButton "Add alarm" clickable=true
|
||||
[3] Button "Timer" clickable=true""",
|
||||
"expect": {"name": "tap", "args": {"index": 2}},
|
||||
"accept_names": ["tap"],
|
||||
},
|
||||
{
|
||||
"id": "wifi_settings",
|
||||
"task": "Turn on Wi-Fi.",
|
||||
"screen": """[0] TextView "Settings" clickable=false
|
||||
[1] TextView "Network & internet" clickable=true
|
||||
[2] TextView "Connected devices" clickable=true
|
||||
[3] TextView "Apps" clickable=true
|
||||
[4] TextView "Battery" clickable=true
|
||||
[5] TextView "Display" clickable=true""",
|
||||
"expect": {"name": "tap", "args": {"index": 1}},
|
||||
"accept_names": ["tap"],
|
||||
},
|
||||
{
|
||||
"id": "wifi_toggle",
|
||||
"task": "Turn on Wi-Fi.",
|
||||
"screen": """[0] TextView "Network & internet" clickable=false
|
||||
[1] Switch "Wi-Fi" checked=false clickable=true
|
||||
[2] TextView "Mobile network" clickable=true
|
||||
[3] Switch "Airplane mode" checked=false clickable=true""",
|
||||
"expect": {"name": "tap", "args": {"index": 1}},
|
||||
"accept_names": ["tap"],
|
||||
},
|
||||
{
|
||||
"id": "search_contact",
|
||||
"task": "Find the contact named Wei Zhang.",
|
||||
"screen": """[0] EditText "Search contacts" clickable=true focused=true
|
||||
[1] TextView "Alice Chen" clickable=true
|
||||
[2] TextView "Bob Lee" clickable=true
|
||||
[3] TextView "Carol Wu" clickable=true""",
|
||||
"expect": {"name": "type_text", "args": {"index": 0, "text": "Wei Zhang"}},
|
||||
"accept_names": ["type_text"],
|
||||
},
|
||||
{
|
||||
"id": "scroll_for_item",
|
||||
"task": "Open the Developer options screen in Settings.",
|
||||
"screen": """[0] TextView "Settings" clickable=false
|
||||
[1] TextView "Display" clickable=true
|
||||
[2] TextView "Sound" clickable=true
|
||||
[3] TextView "Storage" clickable=true
|
||||
[4] TextView "Privacy" clickable=true
|
||||
[5] TextView "Location" clickable=true""",
|
||||
"expect": {"name": "swipe", "args": {"direction": "up"}},
|
||||
"accept_names": ["swipe"],
|
||||
},
|
||||
{
|
||||
"id": "go_back",
|
||||
"task": "Return to the previous screen; you opened the wrong menu.",
|
||||
"screen": """[0] TextView "About phone" clickable=false
|
||||
[1] TextView "Model: Pixel 8" clickable=false
|
||||
[2] TextView "Android version" clickable=true""",
|
||||
"expect": {"name": "press_key", "args": {"key": "back"}},
|
||||
"accept_names": ["press_key"],
|
||||
},
|
||||
{
|
||||
"id": "open_app_by_name",
|
||||
"task": "Open the Camera app.",
|
||||
"screen": """[0] TextView "Swipe up to unlock" clickable=false
|
||||
[1] TextView "10:04" clickable=false""",
|
||||
"expect": {"name": "open_app", "args": {"name": "Camera"}},
|
||||
"accept_names": ["open_app", "swipe"],
|
||||
},
|
||||
{
|
||||
"id": "task_done",
|
||||
"task": "Turn on Wi-Fi.",
|
||||
"screen": """[0] TextView "Network & internet" clickable=false
|
||||
[1] Switch "Wi-Fi" checked=true clickable=true
|
||||
[2] TextView "Connected to HOME-5G" clickable=false""",
|
||||
"expect": {"name": "task_complete", "args": {}},
|
||||
"accept_names": ["task_complete"],
|
||||
},
|
||||
]
|
||||
@@ -0,0 +1,279 @@
|
||||
"""A scored task suite, judged by what the phone shows rather than by what the
|
||||
agent said -- or by what the simulator happens to remember.
|
||||
|
||||
Three things were wrong with the first version of this file, and all three
|
||||
made it score more generously than it claimed to:
|
||||
|
||||
* **``add_alarm`` passed on any typed string containing "7".** Typing ``7``
|
||||
and stopping scored a success on a task whose prompt is "Set an alarm for
|
||||
7:30 AM": no minute, no meridiem, and no requirement that the alarm was ever
|
||||
saved. It now requires a *committed* alarm reading 7:30 AM, which the clock
|
||||
screen lists, so the check is one a real phone can also answer.
|
||||
* **``wifi_on_already`` scored final state only.** Its note said "any tap on
|
||||
the switch fails the task"; the code could not see a tap, so turning Wi-Fi
|
||||
off and back on passed. Switch flips are now recorded and the task fails on
|
||||
any of them -- which is the exact failure W4A8 exhibited (it turned Wi-Fi on
|
||||
and then tapped the switch off again).
|
||||
* **Predicates read ``SimulatedDevice`` attributes**, so every scored run was a
|
||||
simulator run by construction. They read :class:`PhoneState` now, which comes
|
||||
from the UI dump for both backends.
|
||||
|
||||
Repeats are only trials if something differs between them, so each task
|
||||
carries **variants**: the same goal against a renamed control, a reordered
|
||||
screen, or a different starting state. Five runs of one fixture at temperature
|
||||
0 measure repeatability; five variants measure whether the agent can do the
|
||||
task.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import random
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable
|
||||
|
||||
from device import SimElement, SimulatedDevice, demo_phone
|
||||
from phone_state import PhoneState, read_state
|
||||
|
||||
Predicate = Callable[[PhoneState], bool]
|
||||
|
||||
CLOCK_PACKAGE = "com.google.android.deskclock"
|
||||
"""Clock app to reset for the alarm tasks. AOSP builds and several OEMs ship a
|
||||
different package (``com.android.deskclock``, ``com.sec.android.app.clockpackage``
|
||||
on Samsung); override before a real-device run, because clearing the wrong
|
||||
package leaves a stale alarm and the next run scores against it."""
|
||||
|
||||
|
||||
# --------------------------------------------------------------- variations
|
||||
|
||||
def rename(screen: str, old: str, new: str) -> Callable[[SimulatedDevice], None]:
|
||||
"""Same control, different word. Android OEMs really do ship "WLAN"."""
|
||||
def apply(d: SimulatedDevice) -> None:
|
||||
for el in d.screens[screen].elements:
|
||||
if el.label == old:
|
||||
el.label = new
|
||||
return apply
|
||||
|
||||
|
||||
def reorder(screen: str, seed: int) -> Callable[[SimulatedDevice], None]:
|
||||
"""Shuffle the tappable rows. Index-based tapping must not be luck."""
|
||||
def apply(d: SimulatedDevice) -> None:
|
||||
els = d.screens[screen].elements
|
||||
head = [e for e in els if not e.clickable]
|
||||
body = [e for e in els if e.clickable]
|
||||
random.Random(seed).shuffle(body)
|
||||
d.screens[screen].elements = head + body
|
||||
return apply
|
||||
|
||||
|
||||
def decoy(screen: str, label: str) -> Callable[[SimulatedDevice], None]:
|
||||
"""A plausible wrong row, e.g. "Wi-Fi calling" next to "Wi-Fi"."""
|
||||
def apply(d: SimulatedDevice) -> None:
|
||||
d.screens[screen].elements.append(SimElement(label))
|
||||
return apply
|
||||
|
||||
|
||||
def chain(*fns: Callable[[SimulatedDevice], None]) -> Callable[[SimulatedDevice], None]:
|
||||
def apply(d: SimulatedDevice) -> None:
|
||||
for fn in fns:
|
||||
fn(d)
|
||||
return apply
|
||||
|
||||
|
||||
@dataclass
|
||||
class Variant:
|
||||
variant_id: str
|
||||
mutate: Callable[[SimulatedDevice], None] = lambda d: None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- task
|
||||
|
||||
@dataclass
|
||||
class Task:
|
||||
task_id: str
|
||||
prompt: str
|
||||
reaches: Predicate
|
||||
must_not: Predicate = lambda s: False
|
||||
setup: Callable[[SimulatedDevice], None] = lambda d: None
|
||||
max_steps: int = 10
|
||||
note: str = ""
|
||||
variants: list[Variant] = field(default_factory=lambda: [Variant("base")])
|
||||
needs: tuple[str, ...] = ()
|
||||
"""PhoneState facts this task's predicates require that a screen cannot
|
||||
show (``toggled``, ``fields``, ``records``). A task listing any of these
|
||||
cannot be scored on a real phone, and says so instead of scoring wrong."""
|
||||
reset_adb: tuple[tuple[str, ...], ...] = ()
|
||||
"""Commands that put a real phone into this task's starting state. Without
|
||||
them a real-device run is scored against an unknown start, so an empty
|
||||
tuple means "refuse to run me on a phone"."""
|
||||
|
||||
def variant(self, n: int) -> Variant:
|
||||
return self.variants[n % len(self.variants)]
|
||||
|
||||
def build(self, run: int = 0) -> SimulatedDevice:
|
||||
device = demo_phone()
|
||||
self.variant(run).mutate(device)
|
||||
self.setup(device)
|
||||
return device
|
||||
|
||||
def scorable_on(self, state: PhoneState) -> str | None:
|
||||
"""Why this task cannot be scored from ``state``, or None if it can."""
|
||||
if state.source == "device" and self.needs:
|
||||
return (f"needs {', '.join(self.needs)}, which a screen cannot "
|
||||
f"show; simulator only")
|
||||
return None
|
||||
|
||||
def score(self, device_or_state, declared_done: bool, run: int = 0) -> dict:
|
||||
state = (device_or_state if isinstance(device_or_state, PhoneState)
|
||||
else read_state(device_or_state))
|
||||
blocked = self.scorable_on(state)
|
||||
if blocked:
|
||||
return {"task_id": self.task_id, "variant": self.variant(run).variant_id,
|
||||
"scorable": False, "reason": blocked, "success": False,
|
||||
"reached_goal": None, "collateral_damage": None,
|
||||
"declared_done": declared_done}
|
||||
reached = bool(self.reaches(state))
|
||||
damaged = bool(self.must_not(state))
|
||||
return {
|
||||
"task_id": self.task_id,
|
||||
"variant": self.variant(run).variant_id,
|
||||
"scorable": True,
|
||||
"reached_goal": reached,
|
||||
"collateral_damage": damaged,
|
||||
"declared_done": declared_done,
|
||||
# The only row that counts: the phone is in the requested state,
|
||||
# nothing else was broken, and the agent knew to stop.
|
||||
"success": reached and not damaged and declared_done,
|
||||
}
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- oracles
|
||||
|
||||
# Both the on-screen labels and the setting key, because `switch()` matches
|
||||
# either source: "Wi-Fi" normalises onto the `wifi` setting by luck, but
|
||||
# "Airplane mode" does not normalise onto `airplane`, and a name the setting
|
||||
# lookup misses silently falls through to "control not on screen".
|
||||
WIFI = ("Wi-Fi", "WiFi", "WLAN", "wifi")
|
||||
AIRPLANE = ("Airplane mode", "Flight mode", "airplane")
|
||||
|
||||
|
||||
def _wifi_is(want: bool) -> Predicate:
|
||||
# `switch` returns None when the control is not on screen, which is not
|
||||
# the same as off: an agent that never navigated there has not turned
|
||||
# anything on.
|
||||
return lambda s: s.switch(*WIFI) is want
|
||||
|
||||
|
||||
def _airplane_on(s: PhoneState) -> bool:
|
||||
return s.switch(*AIRPLANE) is True
|
||||
|
||||
|
||||
def _alarm_set(hour: int, minute: int, pm: bool) -> Predicate:
|
||||
"""A *saved* alarm reading the requested time.
|
||||
|
||||
The clock screen lists committed alarms, so this reads the screen and works
|
||||
on a real phone. Typing the digits without saving leaves the list empty.
|
||||
"""
|
||||
wanted = f"{hour}:{minute:02d} {'PM' if pm else 'AM'}"
|
||||
return lambda s: s.has_text(wanted)
|
||||
|
||||
|
||||
SUITE: list[Task] = [
|
||||
Task(
|
||||
task_id="wifi_on",
|
||||
prompt="Turn on Wi-Fi.",
|
||||
reaches=_wifi_is(True),
|
||||
must_not=_airplane_on,
|
||||
note="Two screens deep, then a switch. The switch's state is visible, "
|
||||
"so stopping is decidable from the screen.",
|
||||
variants=[
|
||||
Variant("base"),
|
||||
Variant("wlan", rename("network", "Wi-Fi", "WLAN")),
|
||||
Variant("reordered", reorder("network", seed=7)),
|
||||
Variant("decoy", decoy("network", "Wi-Fi calling")),
|
||||
Variant("deep", chain(reorder("settings", seed=3),
|
||||
decoy("network", "Wi-Fi preferences"))),
|
||||
],
|
||||
reset_adb=(("shell", "svc", "wifi", "disable"),),
|
||||
),
|
||||
Task(
|
||||
task_id="wifi_on_already",
|
||||
prompt="Turn on Wi-Fi.",
|
||||
setup=lambda d: d.state.__setitem__("wifi", True),
|
||||
reaches=lambda s: s.switch(*WIFI) is True and s.times_toggled("wifi") == 0,
|
||||
must_not=_airplane_on,
|
||||
note="Already satisfied. The only correct action is task_complete. "
|
||||
"Any flip of the switch fails, including off-and-on again, which "
|
||||
"final state cannot see and which W4A8 actually did.",
|
||||
needs=("toggled",),
|
||||
variants=[
|
||||
Variant("base"),
|
||||
Variant("wlan", rename("network", "Wi-Fi", "WLAN")),
|
||||
Variant("reordered", reorder("network", seed=11)),
|
||||
],
|
||||
),
|
||||
Task(
|
||||
task_id="wifi_off",
|
||||
prompt="Turn off Wi-Fi.",
|
||||
setup=lambda d: d.state.__setitem__("wifi", True),
|
||||
reaches=_wifi_is(False),
|
||||
must_not=_airplane_on,
|
||||
note="The mirror of wifi_on: guards against a model that has simply "
|
||||
"learned to tap switches.",
|
||||
variants=[
|
||||
Variant("base"),
|
||||
Variant("wlan", rename("network", "Wi-Fi", "WLAN")),
|
||||
Variant("reordered", reorder("network", seed=5)),
|
||||
],
|
||||
reset_adb=(("shell", "svc", "wifi", "enable"),),
|
||||
),
|
||||
Task(
|
||||
task_id="airplane_on",
|
||||
prompt="Turn on airplane mode.",
|
||||
reaches=_airplane_on,
|
||||
note="Same screen as Wi-Fi, different row -- catches index confusion, "
|
||||
"which is why one variant reorders that screen.",
|
||||
variants=[
|
||||
Variant("base"),
|
||||
Variant("reordered", reorder("network", seed=2)),
|
||||
Variant("decoy", decoy("network", "Airplane mode settings")),
|
||||
],
|
||||
# `settings put global` needs WRITE_SECURE_SETTINGS, which adb has on a
|
||||
# debuggable build but not on every retail phone; the run fails loudly
|
||||
# if it is refused rather than scoring against an unknown start.
|
||||
reset_adb=(("shell", "settings", "put", "global", "airplane_mode_on", "0"),),
|
||||
),
|
||||
Task(
|
||||
task_id="open_clock",
|
||||
prompt="Open the Clock app and go to the Alarm tab.",
|
||||
reaches=lambda s: s.has_text("Add alarm") or s.has_text("Set time"),
|
||||
note="Navigation only, no state change. Isolates 'can it find an app' "
|
||||
"from 'can it finish'.",
|
||||
variants=[
|
||||
Variant("base"),
|
||||
Variant("reordered", reorder("home", seed=4)),
|
||||
Variant("decoy", decoy("home", "Clock Widget")),
|
||||
],
|
||||
reset_adb=(("shell", "am", "force-stop", CLOCK_PACKAGE),
|
||||
("shell", "input", "keyevent", "KEYCODE_HOME")),
|
||||
),
|
||||
Task(
|
||||
task_id="add_alarm",
|
||||
prompt="Set an alarm for 7:30 AM.",
|
||||
reaches=_alarm_set(7, 30, pm=False),
|
||||
must_not=lambda s: s.has_text("7:30 PM"),
|
||||
max_steps=12,
|
||||
note="The full flow: right app, right tab, add, type both fields, "
|
||||
"leave the meridiem alone, save. Previously this passed on any "
|
||||
"typed string containing '7'.",
|
||||
variants=[
|
||||
Variant("base"),
|
||||
Variant("reordered", reorder("clock", seed=9)),
|
||||
Variant("decoy", decoy("alarms", "Alarm settings")),
|
||||
],
|
||||
# `pm clear` removes saved alarms, so run N does not see run N-1's.
|
||||
reset_adb=(("shell", "pm", "clear", CLOCK_PACKAGE),
|
||||
("shell", "input", "keyevent", "KEYCODE_HOME")),
|
||||
),
|
||||
]
|
||||
|
||||
SUITE_BY_ID = {t.task_id: t for t in SUITE}
|
||||
@@ -0,0 +1,525 @@
|
||||
"""Tests for the Android agent toolkit.
|
||||
|
||||
No phone, emulator or model server is needed: the device is the in-process
|
||||
simulator and the model is a scripted stand-in. That is deliberate -- this
|
||||
host has no `adb`, no emulator and no `/dev/kvm`, so anything that could only
|
||||
be checked against real hardware would otherwise ship unverified.
|
||||
"""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
import agent as agent_mod
|
||||
from agent import AgentConfig, first_tool_call, run_episode
|
||||
from device import demo_phone
|
||||
from executor import execute
|
||||
from ui_tree import parse_ui_dump, render_tree, screen_signature
|
||||
|
||||
# --------------------------------------------------------------------- ui_tree
|
||||
|
||||
REAL_DUMP = """<?xml version="1.0" encoding="UTF-8"?>
|
||||
<hierarchy rotation="0">
|
||||
<node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.android.settings" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][1080,2400]">
|
||||
<node index="1" text="Network & internet" resource-id="android:id/title" class="android.widget.TextView" package="com.android.settings" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,300][1080,460]" />
|
||||
<node index="2" text="" resource-id="" class="android.widget.LinearLayout" package="com.android.settings" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,460][1080,620]" />
|
||||
<node index="3" text="" resource-id="com.android.settings:id/switch_widget" class="android.widget.Switch" package="com.android.settings" content-desc="Wi-Fi" checkable="true" checked="true" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[800,460][1000,620]" />
|
||||
</node>
|
||||
</hierarchy>"""
|
||||
|
||||
|
||||
def test_parses_a_real_dump_and_keeps_only_what_is_actionable():
|
||||
els = parse_ui_dump(REAL_DUMP)
|
||||
labels = [e.text for e in els]
|
||||
assert "Network & internet" in labels # entity decoded
|
||||
assert "Wi-Fi" in labels # fell back to content-desc
|
||||
assert "" not in labels # bare layout node dropped
|
||||
assert [e.index for e in els] == list(range(len(els)))
|
||||
|
||||
|
||||
def test_switch_state_is_visible_to_the_model():
|
||||
"""The one clearly wrong action measured earlier was tapping a switch that
|
||||
was already on; that is only avoidable if the state is on the screen."""
|
||||
rendered = render_tree(parse_ui_dump(REAL_DUMP))
|
||||
assert "checked=true" in rendered
|
||||
assert 'Switch "Wi-Fi"' in rendered
|
||||
|
||||
|
||||
def test_index_maps_to_the_centre_of_the_element():
|
||||
els = parse_ui_dump(REAL_DUMP)
|
||||
wifi = next(e for e in els if e.text == "Wi-Fi")
|
||||
assert wifi.center == (900, 540)
|
||||
|
||||
|
||||
def test_signature_tracks_state_not_pixels():
|
||||
a = parse_ui_dump(REAL_DUMP)
|
||||
shifted = REAL_DUMP.replace("[0,300][1080,460]", "[0,310][1080,470]")
|
||||
assert screen_signature(parse_ui_dump(shifted)) == screen_signature(a)
|
||||
toggled = REAL_DUMP.replace('checkable="true" checked="true"',
|
||||
'checkable="true" checked="false"')
|
||||
assert screen_signature(parse_ui_dump(toggled)) != screen_signature(a)
|
||||
|
||||
|
||||
def test_element_cap_bounds_the_context():
|
||||
big = '<?xml version="1.0"?><hierarchy>' + "".join(
|
||||
f'<node text="row{i}" class="android.widget.TextView" clickable="true" '
|
||||
f'bounds="[0,{i},100,{i + 10}]" />' for i in range(200)
|
||||
) + "</hierarchy>"
|
||||
assert len(parse_ui_dump(big, max_elements=25)) == 25
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- executor
|
||||
|
||||
def test_tap_lands_on_the_right_row_and_reports_prior_state():
|
||||
device = demo_phone()
|
||||
device.current = "network"
|
||||
els = parse_ui_dump(device.dump())
|
||||
wifi = next(e for e in els if e.text == "Wi-Fi")
|
||||
result = execute("tap", {"index": wifi.index}, els, device)
|
||||
assert result.ok and device.state["wifi"] is True
|
||||
assert "was off" in result.message
|
||||
|
||||
|
||||
def test_out_of_range_index_is_a_message_not_an_exception():
|
||||
device = demo_phone()
|
||||
els = parse_ui_dump(device.dump())
|
||||
result = execute("tap", {"index": 99}, els, device)
|
||||
assert not result.ok
|
||||
assert "out of range" in result.message and f"0-{len(els) - 1}" in result.message
|
||||
|
||||
|
||||
def test_bad_arguments_explain_themselves():
|
||||
device = demo_phone()
|
||||
els = parse_ui_dump(device.dump())
|
||||
assert "missing required argument" in execute("tap", {}, els, device).message
|
||||
assert "whole number" in execute("tap", {"index": "banana"}, els, device).message
|
||||
assert "unknown tool" in execute("frobnicate", {}, els, device).message
|
||||
assert "direction must be" in execute("swipe", {"direction": "sideways"}, els, device).message
|
||||
|
||||
|
||||
def test_type_text_focuses_the_field_first():
|
||||
device = demo_phone()
|
||||
device.current = "new_alarm"
|
||||
els = parse_ui_dump(device.dump())
|
||||
field = next(e for e in els if e.text == "Hour")
|
||||
execute("type_text", {"index": field.index, "text": "07"}, els, device)
|
||||
kinds = [a[0] for a in device.actions]
|
||||
assert kinds == ["tap", "text"], kinds
|
||||
assert device.typed == ["07"]
|
||||
|
||||
|
||||
def test_swipe_direction_is_content_movement():
|
||||
device = demo_phone()
|
||||
els = parse_ui_dump(device.dump())
|
||||
execute("swipe", {"direction": "down"}, els, device)
|
||||
_, x1, y1, x2, y2 = device.actions[-1]
|
||||
assert int(y1) > int(y2), "scrolling down must move the finger up"
|
||||
|
||||
|
||||
def test_task_complete_carries_its_summary():
|
||||
device = demo_phone()
|
||||
result = execute("task_complete", {"summary": "Wi-Fi is on"}, [], device)
|
||||
assert result.done and result.summary == "Wi-Fi is on"
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------- agent
|
||||
|
||||
def _scripted(calls):
|
||||
"""A stand-in model that plays a fixed list of tool calls."""
|
||||
seq = list(calls)
|
||||
|
||||
def fake_chat(cfg, messages, *, think):
|
||||
call = seq.pop(0) if seq else None
|
||||
if call is None:
|
||||
return {"choices": [{"message": {"content": "", "tool_calls": []}}]}, 0.01
|
||||
name, args = call
|
||||
return {"choices": [{"message": {"tool_calls": [
|
||||
{"function": {"name": name, "arguments": json.dumps(args)}}
|
||||
]}}]}, 0.01
|
||||
|
||||
return fake_chat
|
||||
|
||||
|
||||
def test_first_tool_call_handles_both_argument_encodings():
|
||||
as_str = {"choices": [{"message": {"tool_calls": [
|
||||
{"function": {"name": "tap", "arguments": '{"index": 2}'}}]}}]}
|
||||
as_dict = {"choices": [{"message": {"tool_calls": [
|
||||
{"function": {"name": "tap", "arguments": {"index": 2}}}]}}]}
|
||||
assert first_tool_call(as_str) == ("tap", {"index": 2})
|
||||
assert first_tool_call(as_dict) == ("tap", {"index": 2})
|
||||
assert first_tool_call({"choices": [{"message": {"content": "hi"}}]}) == (None, {})
|
||||
# Malformed JSON must not kill the episode.
|
||||
broken = {"choices": [{"message": {"tool_calls": [
|
||||
{"function": {"name": "tap", "arguments": "{not json"}}]}}]}
|
||||
assert first_tool_call(broken) == ("tap", {})
|
||||
|
||||
|
||||
def test_completes_a_multi_screen_task(monkeypatch):
|
||||
device = demo_phone()
|
||||
monkeypatch.setattr(agent_mod, "chat", _scripted([
|
||||
("tap", {"index": 3}), # home -> Settings
|
||||
("tap", {"index": 1}), # -> Network & internet
|
||||
("tap", {"index": 1}), # Wi-Fi switch
|
||||
("task_complete", {"summary": "Wi-Fi enabled"}),
|
||||
]))
|
||||
ep = run_episode(device, "Turn on Wi-Fi.", AgentConfig(max_steps=8))
|
||||
assert ep.done and device.state["wifi"] is True
|
||||
assert ep.summary == "Wi-Fi enabled"
|
||||
assert ep.stop_reason == "task_complete"
|
||||
assert ep.n_steps == 4
|
||||
|
||||
|
||||
def test_says_so_when_an_action_changes_nothing(monkeypatch):
|
||||
device = demo_phone()
|
||||
monkeypatch.setattr(agent_mod, "chat", _scripted([
|
||||
("tap", {"index": 0}), # "Phone" on home: goes nowhere
|
||||
("task_complete", {"summary": "done"}),
|
||||
]))
|
||||
ep = run_episode(device, "Do something.", AgentConfig(max_steps=4))
|
||||
assert "did not change" in ep.steps[0].result.message
|
||||
|
||||
|
||||
def test_stops_when_it_repeats_a_no_op(monkeypatch):
|
||||
device = demo_phone()
|
||||
monkeypatch.setattr(agent_mod, "chat", _scripted([("tap", {"index": 0})] * 6))
|
||||
ep = run_episode(device, "Do something.", AgentConfig(max_steps=6))
|
||||
assert not ep.done
|
||||
assert "stuck" in ep.stop_reason
|
||||
assert ep.n_steps < 6, "should give up before the budget"
|
||||
|
||||
|
||||
def test_empty_reply_retries_without_thinking(monkeypatch):
|
||||
"""The measured failure: reasoning never closes `<think>`, so the parser
|
||||
returns nothing. A non-thinking retry is worth ~0.6 s."""
|
||||
seen = []
|
||||
|
||||
def fake_chat(cfg, messages, *, think):
|
||||
seen.append(think)
|
||||
if think:
|
||||
return {"choices": [{"message": {"content": "", "tool_calls": []}}]}, 8.9
|
||||
return {"choices": [{"message": {"tool_calls": [
|
||||
{"function": {"name": "task_complete",
|
||||
"arguments": '{"summary": "fallback"}'}}]}}]}, 0.61
|
||||
|
||||
monkeypatch.setattr(agent_mod, "chat", fake_chat)
|
||||
ep = run_episode(device := demo_phone(), "Task.", AgentConfig(max_steps=3))
|
||||
assert seen == [True, False], seen
|
||||
assert ep.done and ep.summary == "fallback"
|
||||
assert ep.steps[0].thinking is False
|
||||
assert device is not None
|
||||
|
||||
|
||||
def test_gives_up_rather_than_looping_when_no_tool_call_ever_comes(monkeypatch):
|
||||
monkeypatch.setattr(agent_mod, "chat", _scripted([]))
|
||||
ep = run_episode(demo_phone(), "Task.", AgentConfig(max_steps=5))
|
||||
assert not ep.done
|
||||
assert "no tool call" in ep.stop_reason
|
||||
assert ep.n_steps == 1
|
||||
|
||||
|
||||
def test_step_budget_is_enforced(monkeypatch):
|
||||
device = demo_phone()
|
||||
# Alternate two real navigations so nothing is a repeat or a no-op.
|
||||
monkeypatch.setattr(agent_mod, "chat", _scripted(
|
||||
[("tap", {"index": 3}), ("press_key", {"key": "back"})] * 5))
|
||||
ep = run_episode(device, "Wander.", AgentConfig(max_steps=4))
|
||||
assert not ep.done
|
||||
assert ep.n_steps == 4
|
||||
assert "budget exhausted" in ep.stop_reason
|
||||
|
||||
|
||||
def test_transcript_is_readable(monkeypatch):
|
||||
monkeypatch.setattr(agent_mod, "chat", _scripted([
|
||||
("tap", {"index": 3}), ("task_complete", {"summary": "ok"})]))
|
||||
ep = run_episode(demo_phone(), "Open settings.", AgentConfig(max_steps=4))
|
||||
text = ep.transcript()
|
||||
assert "task: Open settings." in text
|
||||
assert "tap(" in text and "task_complete" in text
|
||||
assert "stopped: task_complete" in text
|
||||
|
||||
|
||||
def test_detects_undoing_its_own_action(monkeypatch):
|
||||
"""The measured failure: tapping a switch that is already on, turning it
|
||||
back off, forever. Each tap really does change the screen, so the
|
||||
no-change check cannot see it."""
|
||||
device = demo_phone()
|
||||
device.current = "network"
|
||||
monkeypatch.setattr(agent_mod, "chat", _scripted([("tap", {"index": 1})] * 6))
|
||||
ep = run_episode(device, "Turn on Wi-Fi.", AgentConfig(max_steps=6))
|
||||
assert not ep.done
|
||||
assert "undoing its own action" in ep.stop_reason
|
||||
assert any("undid your previous action" in s.result.message
|
||||
for s in ep.steps if s.result)
|
||||
|
||||
|
||||
def test_normal_navigation_is_not_mistaken_for_oscillation(monkeypatch):
|
||||
"""Opening a submenu and pressing back revisits a screen; that is how
|
||||
phones work and must not trip the guard."""
|
||||
device = demo_phone()
|
||||
monkeypatch.setattr(agent_mod, "chat", _scripted([
|
||||
("tap", {"index": 3}), # home -> settings
|
||||
("press_key", {"key": "back"}), # -> home
|
||||
("tap", {"index": 2}), # home -> clock
|
||||
("press_key", {"key": "back"}), # -> home
|
||||
("task_complete", {"summary": "looked around"}),
|
||||
]))
|
||||
ep = run_episode(device, "Look around.", AgentConfig(max_steps=8))
|
||||
assert ep.done, ep.stop_reason
|
||||
assert ep.n_steps == 5
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ task suite
|
||||
|
||||
def test_suite_scores_on_device_state_not_on_what_the_agent_said():
|
||||
"""The failure this model exhibits is claiming completion it did not
|
||||
achieve, so the score must not read the transcript."""
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["wifi_on"]
|
||||
device = task.build()
|
||||
assert task.score(device, declared_done=True)["success"] is False
|
||||
device.state["wifi"] = True
|
||||
assert task.score(device, declared_done=True)["success"] is True
|
||||
# Reaching the goal without stopping is not success either.
|
||||
assert task.score(device, declared_done=False)["success"] is False
|
||||
|
||||
|
||||
def test_collateral_damage_fails_the_task():
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["wifi_on"]
|
||||
device = task.build()
|
||||
device.state.update(wifi=True, airplane=True)
|
||||
score = task.score(device, declared_done=True)
|
||||
assert score["reached_goal"] and score["collateral_damage"]
|
||||
assert score["success"] is False
|
||||
|
||||
|
||||
def test_already_satisfied_task_starts_satisfied():
|
||||
"""Its only correct action is task_complete; tapping the switch fails."""
|
||||
from phone_state import read_state
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["wifi_on_already"]
|
||||
device = task.build()
|
||||
assert task.reaches(read_state(device))
|
||||
assert task.score(device, declared_done=True)["success"] is True
|
||||
|
||||
|
||||
def test_already_satisfied_task_fails_on_a_switch_round_trip():
|
||||
"""W4A8 turned Wi-Fi on and then tapped the switch off again. Final state
|
||||
cannot see that, so this task scored it a pass until the flips were
|
||||
recorded."""
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["wifi_on_already"]
|
||||
device = task.build()
|
||||
device.launch("Settings")
|
||||
device.tap(540, 360) # Network & internet
|
||||
device.tap(540, 360) # Wi-Fi off
|
||||
device.tap(540, 360) # ... and on again
|
||||
assert device.state["wifi"] is True # final state looks correct
|
||||
assert task.score(device, declared_done=True)["success"] is False
|
||||
|
||||
|
||||
def test_goal_is_read_from_the_setting_not_from_the_final_screen():
|
||||
"""An agent that turns Wi-Fi on and then goes Home has still done it."""
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["wifi_on"]
|
||||
device = task.build()
|
||||
device.launch("Settings")
|
||||
device.tap(540, 360)
|
||||
device.tap(540, 360)
|
||||
device.keyevent("home")
|
||||
assert task.score(device, declared_done=True)["success"] is True
|
||||
|
||||
|
||||
def test_never_finding_the_screen_is_not_scored_as_off():
|
||||
"""`switch()` returns None when neither the setting nor the screen knows
|
||||
the control; a predicate must not read that as 'off' and pass a task."""
|
||||
from phone_state import PhoneState
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
empty = PhoneState(source="device")
|
||||
assert empty.switch("Wi-Fi") is None
|
||||
assert SUITE_BY_ID["wifi_off"].score(empty, declared_done=True)["success"] is False
|
||||
|
||||
|
||||
def test_suite_includes_a_mirror_task_for_switch_tapping():
|
||||
"""Guards against a model that has simply learned 'tap the switch'."""
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
from phone_state import read_state
|
||||
|
||||
off = SUITE_BY_ID["wifi_off"]
|
||||
device = off.build()
|
||||
assert device.state["wifi"] is True
|
||||
assert not off.reaches(read_state(device))
|
||||
device.state["wifi"] = False
|
||||
assert off.reaches(read_state(device))
|
||||
|
||||
|
||||
def test_chat_retries_transient_failures(monkeypatch):
|
||||
"""A phone task is a dozen round trips; one dropped connection should cost
|
||||
a retry, not the episode."""
|
||||
import agent as mod
|
||||
from urllib import error as urlerror
|
||||
|
||||
calls = {"n": 0}
|
||||
|
||||
def flaky(req, timeout=None):
|
||||
calls["n"] += 1
|
||||
if calls["n"] < 3:
|
||||
raise urlerror.URLError("connection reset")
|
||||
class _R:
|
||||
def __enter__(self): return self
|
||||
def __exit__(self, *a): return False
|
||||
def read(self): return b'{"choices":[{"message":{"tool_calls":[]}}]}'
|
||||
return _R()
|
||||
|
||||
monkeypatch.setattr(mod.urlrequest, "urlopen", flaky)
|
||||
monkeypatch.setattr(mod.time, "sleep", lambda s: None)
|
||||
cfg = AgentConfig(retries=3)
|
||||
body, _ = mod.chat(cfg, [{"role": "user", "content": "x"}], think=False)
|
||||
assert calls["n"] == 3 and "choices" in body
|
||||
|
||||
|
||||
def test_chat_gives_up_with_a_clear_error(monkeypatch):
|
||||
import agent as mod
|
||||
from urllib import error as urlerror
|
||||
|
||||
monkeypatch.setattr(mod.urlrequest, "urlopen",
|
||||
lambda *a, **k: (_ for _ in ()).throw(urlerror.URLError("down")))
|
||||
monkeypatch.setattr(mod.time, "sleep", lambda s: None)
|
||||
with pytest.raises(mod.ChatError, match="after 2 attempts"):
|
||||
mod.chat(AgentConfig(retries=2), [{"role": "user", "content": "x"}], think=False)
|
||||
|
||||
|
||||
# --------------------------------------------------- oracles that were wrong
|
||||
|
||||
def test_alarm_task_needs_a_saved_alarm_not_a_typed_digit():
|
||||
"""This oracle used to pass on any typed string containing "7", so typing
|
||||
`7` and stopping scored a success on "Set an alarm for 7:30 AM"."""
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["add_alarm"]
|
||||
|
||||
typed_only = task.build()
|
||||
typed_only.launch("Clock")
|
||||
typed_only.tap(540, 200) # Alarm tab
|
||||
typed_only.tap(540, 360) # Add alarm
|
||||
typed_only.tap(540, 360)
|
||||
typed_only.input_text("7")
|
||||
assert task.score(typed_only, declared_done=True)["success"] is False
|
||||
|
||||
both_fields = task.build()
|
||||
both_fields.launch("Clock")
|
||||
both_fields.tap(540, 200)
|
||||
both_fields.tap(540, 360)
|
||||
both_fields.tap(540, 360)
|
||||
both_fields.input_text("7")
|
||||
both_fields.tap(540, 520)
|
||||
both_fields.input_text("30")
|
||||
assert task.score(both_fields, declared_done=True)["success"] is False, \
|
||||
"typing both fields without saving is not a set alarm"
|
||||
|
||||
both_fields.tap(540, 840) # Save
|
||||
assert task.score(both_fields, declared_done=True)["success"] is True
|
||||
|
||||
|
||||
def test_alarm_task_rejects_the_wrong_meridiem():
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["add_alarm"]
|
||||
d = task.build()
|
||||
d.launch("Clock"); d.tap(540, 200); d.tap(540, 360)
|
||||
d.tap(540, 360); d.input_text("7")
|
||||
d.tap(540, 520); d.input_text("30")
|
||||
d.tap(540, 680) # AM -> PM
|
||||
d.tap(540, 840) # Save
|
||||
score = task.score(d, declared_done=True)
|
||||
assert score["success"] is False and score["collateral_damage"] is True
|
||||
|
||||
|
||||
def test_a_saved_alarm_is_visible_on_the_screen():
|
||||
"""The oracle reads the clock app's list, so it works on a real phone."""
|
||||
from phone_state import read_state
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
d = SUITE_BY_ID["add_alarm"].build()
|
||||
d.launch("Clock"); d.tap(540, 200); d.tap(540, 360)
|
||||
d.tap(540, 360); d.input_text("7")
|
||||
d.tap(540, 520); d.input_text("30")
|
||||
d.tap(540, 840)
|
||||
assert read_state(d).has_text("7:30 AM")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ variants
|
||||
|
||||
def test_every_task_has_variants_so_repeats_are_trials():
|
||||
"""Five runs of one fixture at temperature 0 measure repeatability. The
|
||||
suite has to vary something for a repeat to be a trial."""
|
||||
from tasks import SUITE
|
||||
|
||||
for task in SUITE:
|
||||
assert len(task.variants) >= 1
|
||||
ids = [v.variant_id for v in task.variants]
|
||||
assert len(ids) == len(set(ids)), task.task_id
|
||||
assert any(len(t.variants) > 1 for t in SUITE)
|
||||
|
||||
|
||||
def test_variants_change_the_phone_but_not_the_goal():
|
||||
"""A renamed or reordered control is still the same task."""
|
||||
from phone_state import read_state
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["wifi_on"]
|
||||
labels_seen = set()
|
||||
for run in range(len(task.variants)):
|
||||
d = task.build(run)
|
||||
d.launch("Settings")
|
||||
d.tap(540, 360)
|
||||
labels_seen.add(read_state(d).source)
|
||||
# The goal predicate is unchanged; only the route differs.
|
||||
assert task.score(d, declared_done=True)["success"] is False
|
||||
d.state["wifi"] = True
|
||||
assert task.score(d, declared_done=True)["success"] is True
|
||||
assert labels_seen == {"sim"}
|
||||
|
||||
|
||||
def test_wlan_variant_renames_the_control_the_agent_must_find():
|
||||
from phone_state import read_state
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
task = SUITE_BY_ID["wifi_on"]
|
||||
variant = next(i for i, v in enumerate(task.variants) if v.variant_id == "wlan")
|
||||
d = task.build(variant)
|
||||
d.launch("Settings"); d.tap(540, 360)
|
||||
assert read_state(d).has_text("WLAN")
|
||||
assert not read_state(d).has_text("Wi-Fi")
|
||||
|
||||
|
||||
# -------------------------------------------------- real device honesty
|
||||
|
||||
def test_history_dependent_task_refuses_to_score_on_a_real_phone():
|
||||
"""A phone cannot report which switches were flipped, so the task says so
|
||||
instead of scoring from final state and calling a round trip a pass."""
|
||||
from phone_state import PhoneState
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
state = PhoneState(source="device", settings={"wifi": True})
|
||||
score = SUITE_BY_ID["wifi_on_already"].score(state, declared_done=True)
|
||||
assert score["scorable"] is False
|
||||
assert "simulator only" in score["reason"]
|
||||
assert score["success"] is False
|
||||
|
||||
|
||||
def test_tasks_runnable_on_a_phone_declare_how_to_reset_it():
|
||||
"""Without a reset the run is scored against an unknown starting state."""
|
||||
from tasks import SUITE_BY_ID
|
||||
|
||||
assert SUITE_BY_ID["wifi_on"].reset_adb
|
||||
assert SUITE_BY_ID["wifi_off"].reset_adb
|
||||
# and the sim-only one is not pretending otherwise
|
||||
assert SUITE_BY_ID["wifi_on_already"].needs == ("toggled",)
|
||||
@@ -0,0 +1,155 @@
|
||||
"""Recover a tool call from what the model actually emitted.
|
||||
|
||||
vLLM's stock `glm45` parser handles Spark's tool output most of the time --
|
||||
4 of 5 screens in a spot check -- and then does not. On the Wi-Fi toggle
|
||||
screen, the one where a switch's state decides the right action, the model
|
||||
dropped out of GLM-4.5 format entirely and emitted
|
||||
|
||||
[tap]
|
||||
{"tool": "tap", "parameters": {"index": 1}
|
||||
|
||||
with the JSON unterminated. `tool_calls` comes back empty, and an agent that
|
||||
only reads `tool_calls` sees a step with no action at all.
|
||||
|
||||
For a phone agent that is the difference between finishing a task and
|
||||
stopping in the middle of it, so this module tries, in order:
|
||||
|
||||
1. the parsed `tool_calls` (the normal path -- nothing here second-guesses it),
|
||||
2. a GLM-4.5 `<tool_call>` block left in the content,
|
||||
3. the bracketed `[name]` + JSON form above, repairing unbalanced braces,
|
||||
4. a bare JSON object naming a tool.
|
||||
|
||||
Every recovery is reported, because an agent that quietly repairs its model's
|
||||
output will hide a regression in that model.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
_BRACKET = re.compile(r"\[([a-z_][a-z0-9_]*)\]", re.I)
|
||||
_GLM_CALL = re.compile(
|
||||
r"<tool_call>\s*([a-z_][a-z0-9_]*)(.*?)</tool_call>", re.I | re.S)
|
||||
_GLM_ARG = re.compile(
|
||||
r"<arg_key>(.*?)</arg_key>\s*<arg_value>(.*?)</arg_value>", re.I | re.S)
|
||||
|
||||
_ARG_CONTAINERS = ("parameters", "arguments", "args")
|
||||
_NAME_KEYS = ("tool", "name", "tool_name", "function")
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedCall:
|
||||
name: str | None
|
||||
args: dict
|
||||
source: str
|
||||
"""Which path produced it: tool_calls | glm_block | bracketed | bare_json."""
|
||||
|
||||
@property
|
||||
def recovered(self) -> bool:
|
||||
return self.name is not None and self.source != "tool_calls"
|
||||
|
||||
|
||||
def _coerce(value: str):
|
||||
"""Tool arguments arrive as strings; indices must become integers."""
|
||||
text = value.strip()
|
||||
if re.fullmatch(r"-?\d+", text):
|
||||
return int(text)
|
||||
if text.lower() in ("true", "false"):
|
||||
return text.lower() == "true"
|
||||
return text
|
||||
|
||||
|
||||
def _balance(text: str) -> str:
|
||||
"""Close a JSON object the model stopped short of finishing."""
|
||||
depth_curly = text.count("{") - text.count("}")
|
||||
depth_square = text.count("[") - text.count("]")
|
||||
if depth_curly < 0 or depth_square < 0:
|
||||
return text
|
||||
return text + "]" * depth_square + "}" * depth_curly
|
||||
|
||||
|
||||
def _load(text: str) -> dict | None:
|
||||
for candidate in (text, _balance(text)):
|
||||
try:
|
||||
value = json.loads(candidate)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(value, dict):
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _unwrap(payload: dict, fallback_name: str | None) -> tuple[str | None, dict]:
|
||||
"""Pull a (name, args) pair out of assorted self-describing shapes."""
|
||||
name = fallback_name
|
||||
for key in _NAME_KEYS:
|
||||
value = payload.get(key)
|
||||
if isinstance(value, str) and value:
|
||||
name = value
|
||||
break
|
||||
if isinstance(value, dict) and isinstance(value.get("name"), str):
|
||||
name = value["name"]
|
||||
payload = {**payload, **value}
|
||||
break
|
||||
args: dict = {}
|
||||
for key in _ARG_CONTAINERS:
|
||||
inner = payload.get(key)
|
||||
if isinstance(inner, dict):
|
||||
args = dict(inner)
|
||||
break
|
||||
if isinstance(inner, str):
|
||||
parsed = _load(inner)
|
||||
if parsed is not None:
|
||||
args = parsed
|
||||
break
|
||||
if not args:
|
||||
args = {k: v for k, v in payload.items()
|
||||
if k not in _NAME_KEYS + _ARG_CONTAINERS}
|
||||
return name, args
|
||||
|
||||
|
||||
def _first_json_object(text: str) -> str | None:
|
||||
start = text.find("{")
|
||||
return text[start:] if start >= 0 else None
|
||||
|
||||
|
||||
def extract_tool_call(message: dict) -> ParsedCall:
|
||||
"""The one call to act on, from whichever shape the model used."""
|
||||
calls = message.get("tool_calls") or []
|
||||
if calls:
|
||||
fn = calls[0].get("function", {}) or {}
|
||||
raw = fn.get("arguments")
|
||||
if isinstance(raw, dict):
|
||||
args = raw
|
||||
else:
|
||||
args = _load(raw or "{}") or {}
|
||||
if set(args) & set(_ARG_CONTAINERS):
|
||||
_, args = _unwrap(args, fn.get("name"))
|
||||
return ParsedCall(fn.get("name"), args, "tool_calls")
|
||||
|
||||
content = message.get("content") or ""
|
||||
reasoning = message.get("reasoning_content") or ""
|
||||
for text in (content, reasoning):
|
||||
if not text.strip():
|
||||
continue
|
||||
|
||||
if m := _GLM_CALL.search(text):
|
||||
args = {k.strip(): _coerce(v) for k, v in _GLM_ARG.findall(m.group(2))}
|
||||
return ParsedCall(m.group(1), args, "glm_block")
|
||||
|
||||
bracket = _BRACKET.search(text)
|
||||
blob = _first_json_object(text)
|
||||
if blob is not None:
|
||||
payload = _load(blob)
|
||||
if payload is not None:
|
||||
name, args = _unwrap(payload, bracket.group(1) if bracket else None)
|
||||
if name:
|
||||
return ParsedCall(name, args, "bracketed" if bracket else "bare_json")
|
||||
|
||||
if bracket and not blob:
|
||||
# A naked "[press_key]" with no arguments is still an instruction.
|
||||
return ParsedCall(bracket.group(1), {}, "bracketed")
|
||||
|
||||
return ParsedCall(None, {}, "none")
|
||||
@@ -0,0 +1,146 @@
|
||||
"""Turn a uiautomator dump into what a text-only agent can act on.
|
||||
|
||||
Spark-X2.5 never sees a screenshot, so the whole agent hinges on this file:
|
||||
a screen has to arrive as a short, stable, indexed list of elements, and an
|
||||
index the model returns has to map back to a point on the glass.
|
||||
|
||||
Two decisions shape the format, both forced by the model rather than by taste:
|
||||
|
||||
* **Filter hard.** A real settings screen dumps 150+ nodes, most of them
|
||||
layout containers. At 56 KiB/token of KV cache and a 1.7B model that both
|
||||
costs context and buries the answer, so only nodes that are interactive or
|
||||
carry text survive.
|
||||
* **Index, do not coordinate.** The model is asked for `index: 3`, never for
|
||||
pixels -- a text model has no way to know where 3 is on the glass, and
|
||||
coordinates in the transcript would be a source of silent nonsense. The
|
||||
mapping back to a centre point happens here, where the bounds are known.
|
||||
|
||||
The rendered form is the one the scenario fixtures were written in, so the
|
||||
decision benchmark and the live agent observe identically.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import xml.etree.ElementTree as ET
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
_BOUNDS = re.compile(r"\[(-?\d+),(-?\d+)\]\[(-?\d+),(-?\d+)\]")
|
||||
|
||||
# Attributes worth showing when they are true, in the order a reader wants
|
||||
# them. `checked` earns its place: the one clearly wrong action in the earlier
|
||||
# measurements was tapping a Wi-Fi switch that was already on, which is only
|
||||
# avoidable if the state is on the screen the model sees.
|
||||
_FLAGS = ("clickable", "checked", "selected", "scrollable", "focused")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Element:
|
||||
index: int
|
||||
cls: str
|
||||
text: str
|
||||
resource_id: str = ""
|
||||
package: str = ""
|
||||
bounds: tuple[int, int, int, int] = (0, 0, 0, 0)
|
||||
flags: dict[str, bool] = field(default_factory=dict)
|
||||
editable: bool = False
|
||||
|
||||
@property
|
||||
def center(self) -> tuple[int, int]:
|
||||
x1, y1, x2, y2 = self.bounds
|
||||
return (x1 + x2) // 2, (y1 + y2) // 2
|
||||
|
||||
@property
|
||||
def short_class(self) -> str:
|
||||
return self.cls.rsplit(".", 1)[-1] if self.cls else "View"
|
||||
|
||||
def render(self) -> str:
|
||||
parts = [f"[{self.index}]", self.short_class, f'"{self.text}"']
|
||||
for flag in _FLAGS:
|
||||
if flag in self.flags:
|
||||
parts.append(f"{flag}={str(self.flags[flag]).lower()}")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def _truthy(node: ET.Element, name: str) -> bool:
|
||||
return node.get(name, "false") == "true"
|
||||
|
||||
|
||||
def _label(node: ET.Element) -> str:
|
||||
"""What to call this element, preferring what a person would read."""
|
||||
for attr in ("text", "content-desc"):
|
||||
value = (node.get(attr) or "").strip()
|
||||
if value:
|
||||
return value.replace("\n", " ")
|
||||
# A bare resource id is a poor label but far better than an empty string
|
||||
# when the element is the only way forward (unlabelled icon buttons).
|
||||
rid = node.get("resource-id") or ""
|
||||
return rid.rsplit("/", 1)[-1] if rid else ""
|
||||
|
||||
|
||||
def _interesting(node: ET.Element) -> bool:
|
||||
if any(_truthy(node, a) for a in ("clickable", "checkable", "scrollable",
|
||||
"long-clickable", "focusable")):
|
||||
return True
|
||||
return bool(_label(node)) and not _truthy(node, "password")
|
||||
|
||||
|
||||
def parse_bounds(raw: str | None) -> tuple[int, int, int, int]:
|
||||
m = _BOUNDS.search(raw or "")
|
||||
if not m:
|
||||
return (0, 0, 0, 0)
|
||||
return tuple(int(g) for g in m.groups()) # type: ignore[return-value]
|
||||
|
||||
|
||||
def parse_ui_dump(xml_text: str, *, max_elements: int = 60) -> list[Element]:
|
||||
"""uiautomator XML -> the elements worth showing, in reading order."""
|
||||
root = ET.fromstring(xml_text)
|
||||
elements: list[Element] = []
|
||||
for node in root.iter("node"):
|
||||
if not _interesting(node):
|
||||
continue
|
||||
label = _label(node)
|
||||
clickable = _truthy(node, "clickable")
|
||||
# An unlabelled, non-interactive node tells the model nothing it can
|
||||
# use and costs it context.
|
||||
if not label and not clickable and not _truthy(node, "checkable"):
|
||||
continue
|
||||
flags = {"clickable": clickable}
|
||||
if _truthy(node, "checkable"):
|
||||
flags["checked"] = _truthy(node, "checked")
|
||||
if _truthy(node, "selected"):
|
||||
flags["selected"] = True
|
||||
if _truthy(node, "scrollable"):
|
||||
flags["scrollable"] = True
|
||||
if _truthy(node, "focused"):
|
||||
flags["focused"] = True
|
||||
elements.append(
|
||||
Element(
|
||||
index=len(elements),
|
||||
cls=node.get("class", ""),
|
||||
text=label,
|
||||
resource_id=node.get("resource-id", ""),
|
||||
package=node.get("package", ""),
|
||||
bounds=parse_bounds(node.get("bounds")),
|
||||
flags=flags,
|
||||
editable="EditText" in (node.get("class") or ""),
|
||||
)
|
||||
)
|
||||
if len(elements) >= max_elements:
|
||||
break
|
||||
return elements
|
||||
|
||||
|
||||
def render_tree(elements: list[Element]) -> str:
|
||||
"""The exact text the model is shown."""
|
||||
return "\n".join(e.render() for e in elements)
|
||||
|
||||
|
||||
def screen_signature(elements: list[Element]) -> str:
|
||||
"""A cheap identity for a screen, to notice when an action changed nothing.
|
||||
|
||||
Labels and flags only: coordinates shift with animations and scroll offsets,
|
||||
and treating those as a new screen would hide a stuck agent.
|
||||
"""
|
||||
return "|".join(f"{e.short_class}:{e.text}:{e.flags.get('checked', '')}"
|
||||
for e in elements)
|
||||
Reference in new issue
Block a user