Screenshots or the accessibility tree

Every computer-use stack argues about this: look at the pixels, or read the element tree. We ran the same ten-step desktop task ten times on a Kubi screen, letting the agent use only screenshots, only the tree, or both, and measured all of it. The short answer is that neither is faster, and that almost all the lost time came from somewhere else entirely.

Our own lab, 2026-11-1510 runs, 8 checks each, all passedClaude Opus 5.5 through Kubi's MCPUpdated 2026-11-15

The task and the setup

One task, the same every run: open Chromium and an incognito window, search two engines and keep their answers, close the browser, type a four-column table into LibreOffice Calc, save it as CSV to a path, show it in a terminal with cat, and open the file manager on that folder. Eight checks had to pass, read back through the API rather than claimed by the agent. All ten runs passed all eight.

  • The agent was a fresh non-interactive session each run, on Claude Opus 5.5, allowed only the lab's MCP tools: no shell of its own, no file access outside them.
  • The modes differ by one paragraph of the prompt and by which tools were denied. Screenshots only: no read_tree and no element actions. Tree only: no screenshot, no zoom, no wait. Both: everything.
  • The screen was reset between runs, apps closed and the folder emptied, and the result was checked through the API.

What the runs cost

Wall time, and where it went. Model time and tool time add up to the total.

ModeReaderTime sTurnsTokens inImages
Screenshotsbefore the fixes281301.80M26
Screenshotsbefore the fixes335331.94M29
Treebefore the fixes375523.43M0
Treeafter two fixes301341.84M0
Treeafter two fixes324392.54M0
Bothafter two fixes314311.85M22
Bothafter two fixes328311.86M23

Seven of the ten runs. The two fixes came out of the earlier runs: prose names were being cut at 200 characters, and a resumed screen kept an older daemon.

With the reader fixed, the three modes are level: screenshots 281 to 335 seconds, the tree 301 to 324, both 314 to 328. The screenshot runs alone spread 19% between themselves, which is wider than the gap between the modes. On this task, no mode is faster.

The tree does change the shape of the bill rather than its size: zero images, and after the fix about the same tokens in. Before the fix the tree runs cost nearly twice the tokens, because a truncated name sent the agent round again. If you are choosing on price, measure your own reader before you trust a rule of thumb.

Where the turns went

Turns lost against the shortest path, over the ten runs, by cause.

CauseRunsTurns lostWhose fault
Curly apostrophes from LibreOffice AutoCorrect10 of 10about 3 eachour image's defaults
Calc's CSV character set not in the tree4 of 4 tree runsabout 5 eachour tree reader
An answer cut at 200 characters2 of 2 old runs3 to 6our reader, since fixed
key written for keys3 of 101our MCP schema
An invented action that does not exist3 of 101our MCP schema
More than 20 actions in one call3 of 101 to 2our MCP schema

Every row is ours. Not one lost turn was the model failing to understand the desktop.

This is the part worth taking away. The argument about pixels against trees is mostly noise next to an application's own defaults, a tool schema that invites a wrong field name, and a reader that drops the one value the agent needed. About 60% of each run was model turns and 40% tools, and most of the tool time was the agent's own fixed waits: 59 to 82 seconds a run, in every mode.

How to choose, in practice

  • Use the tree to find and act on a control you can name: menus, fields, buttons, lists. It is steadier than aiming at coordinates.
  • Use a screenshot to check the result, and whenever the app draws something the tree does not expose, which on this task was a character-set dropdown.
  • Give the agent both when you can. It costs little and removes the failure where one of them is blind.
  • Fix the dull things first: your image's autocorrect, your tool schema's field names, and the waits your agent inserts out of habit.
  • Measure a task you care about. Ten runs of one task is enough to see a 19% spread, which is also enough to warn you off a single run.

Questions people ask

Is the accessibility tree faster than screenshots?

Not on this task. With our reader fixed, the tree ran 301 to 324 seconds and screenshots 281 to 335, inside each other's spread. The tree sends no images, and after the fix cost about the same tokens.

Which should my agent use on a Kubi screen?

Both, if your agent can. read_tree gives elements with ids you can act on, screenshot gives the picture, and act takes either. The docs describe the loop and what each tool returns.

Can I see this study in full?

The write-up lives with Kubi's other research, including the harness, the eight checks and every run's numbers. The summary here is the part that is useful without the source.

Give your agent a desktop that stays on.

Private beta. Founding price $20 a screen a month, kept while you stay.

Get early access