Screenshots or the accessibility tree
Every computer-use stack argues about this: look at the pixels, or read the element tree. We ran the same ten-step desktop task ten times on a Kubi screen, letting the agent use only screenshots, only the tree, or both, and measured all of it. The short answer is that neither is faster, and that almost all the lost time came from somewhere else entirely.
The task and the setup
One task, the same every run: open Chromium and an incognito window, search two engines and keep their answers, close the browser, type a four-column table into LibreOffice Calc, save it as CSV to a path, show it in a terminal with cat, and open the file manager on that folder. Eight checks had to pass, read back through the API rather than claimed by the agent. All ten runs passed all eight.
- The agent was a fresh non-interactive session each run, on Claude Opus 5.5, allowed only the lab's MCP tools: no shell of its own, no file access outside them.
- The modes differ by one paragraph of the prompt and by which tools were denied. Screenshots only: no read_tree and no element actions. Tree only: no screenshot, no zoom, no wait. Both: everything.
- The screen was reset between runs, apps closed and the folder emptied, and the result was checked through the API.
What the runs cost
Wall time, and where it went. Model time and tool time add up to the total.
| Mode | Reader | Time s | Turns | Tokens in | Images |
|---|---|---|---|---|---|
| Screenshots | before the fixes | 281 | 30 | 1.80M | 26 |
| Screenshots | before the fixes | 335 | 33 | 1.94M | 29 |
| Tree | before the fixes | 375 | 52 | 3.43M | 0 |
| Tree | after two fixes | 301 | 34 | 1.84M | 0 |
| Tree | after two fixes | 324 | 39 | 2.54M | 0 |
| Both | after two fixes | 314 | 31 | 1.85M | 22 |
| Both | after two fixes | 328 | 31 | 1.86M | 23 |
Seven of the ten runs. The two fixes came out of the earlier runs: prose names were being cut at 200 characters, and a resumed screen kept an older daemon.
With the reader fixed, the three modes are level: screenshots 281 to 335 seconds, the tree 301 to 324, both 314 to 328. The screenshot runs alone spread 19% between themselves, which is wider than the gap between the modes. On this task, no mode is faster.
The tree does change the shape of the bill rather than its size: zero images, and after the fix about the same tokens in. Before the fix the tree runs cost nearly twice the tokens, because a truncated name sent the agent round again. If you are choosing on price, measure your own reader before you trust a rule of thumb.
Where the turns went
Turns lost against the shortest path, over the ten runs, by cause.
| Cause | Runs | Turns lost | Whose fault |
|---|---|---|---|
| Curly apostrophes from LibreOffice AutoCorrect | 10 of 10 | about 3 each | our image's defaults |
| Calc's CSV character set not in the tree | 4 of 4 tree runs | about 5 each | our tree reader |
| An answer cut at 200 characters | 2 of 2 old runs | 3 to 6 | our reader, since fixed |
| key written for keys | 3 of 10 | 1 | our MCP schema |
| An invented action that does not exist | 3 of 10 | 1 | our MCP schema |
| More than 20 actions in one call | 3 of 10 | 1 to 2 | our MCP schema |
Every row is ours. Not one lost turn was the model failing to understand the desktop.
This is the part worth taking away. The argument about pixels against trees is mostly noise next to an application's own defaults, a tool schema that invites a wrong field name, and a reader that drops the one value the agent needed. About 60% of each run was model turns and 40% tools, and most of the tool time was the agent's own fixed waits: 59 to 82 seconds a run, in every mode.
How to choose, in practice
- Use the tree to find and act on a control you can name: menus, fields, buttons, lists. It is steadier than aiming at coordinates.
- Use a screenshot to check the result, and whenever the app draws something the tree does not expose, which on this task was a character-set dropdown.
- Give the agent both when you can. It costs little and removes the failure where one of them is blind.
- Fix the dull things first: your image's autocorrect, your tool schema's field names, and the waits your agent inserts out of habit.
- Measure a task you care about. Ten runs of one task is enough to see a 19% spread, which is also enough to warn you off a single run.
Questions people ask
Is the accessibility tree faster than screenshots?
Which should my agent use on a Kubi screen?
Can I see this study in full?
Read next
The docs hold the reference; these pages hold what we measured.










