A benchmark on a hundred screens
Evaluating a computer-use agent means running the same task many times on identical machines, and the machines are usually the reason people do not. A disposable screen is billed by the minute and created from your own template, so a hundred of them for half an hour is $1.75 of machine time.
The arithmetic first
A disposable screen is $0.035 an hour while it runs, charged by the minute, and $0.003 an hour stopped. Running a hundred screens for half an hour costs the same as one screen for fifty hours: $1.75. A thousand five-minute runs are $2.92. A thousand half-hour runs are $17.50. Parallelism changes how long you wait, not what you pay.
Those numbers are the machines only. The model tokens are yours and will dwarf them: on the public Cua Driver Bench data, one trial of one ten-task desktop benchmark cost about $0.40 to $0.60 in model spend with a mid-sized model. The machine is the cheap part of an eval, which is not what most people assume.
Make the machines identical
- Build one screen the way the run needs it, with the app, the fixtures and the settings in place.
- Save it as a template, and start every run's screen from that template. Each organization gets five templates, and each screen three snapshots.
- Snapshot before a destructive step so a failed trial can be reset rather than rebuilt.
- Keep the image's own defaults in mind: in our own tree eval, LibreOffice's autocorrect cost about three turns in every single run. An eval measures your harness as much as your agent.
Drive the fleet from your own code
The API creates, starts, stops and deletes screens; the hosted MCP server deliberately cannot, so an agent cannot spin up machines on your account. A benchmark runner is therefore a plain script with an API key: create from the template, run the trial, read the result back through the API rather than trusting the agent's report, delete the screen.
- Write calls take an idempotency key, so a retry after a timeout does not run the trial twice.
- Name the run with a header and the calls group under one label instead of a flat log.
- Check the outcome on the machine, not in the transcript. Our own eval passed or failed each run on eight checks read through the API.
- Each organization has a cap on how many screens can exist at once, so ask before you plan a thousand in parallel.
What we learned running one
Our tree eval was ten runs of one task, which is small, and it still produced a 19% spread between two runs of the identical setup. Single runs prove nothing. It also showed where the time really goes: about 60% model turns, about 40% tools, and most of the tool time was the agent's own fixed waits, 59 to 82 seconds a run. If you are optimising an eval, start there rather than with the machines.
And hold one thing fixed at a time. Most published computer-use leaderboards vary the model between rows, which is why they disagree with each other; the one dataset that holds the model steady reorders the drivers.
Questions people ask
What does it cost to run 1,000 computer-use trials?
Can I run hundreds of screens at once?
Can an agent create its own screens for a run?
Do you publish your own benchmark results?
Read next
The docs hold the reference; these pages hold what we measured.










