A benchmark on a hundred screens

Evaluating a computer-use agent means running the same task many times on identical machines, and the machines are usually the reason people do not. A disposable screen is billed by the minute and created from your own template, so a hundred of them for half an hour is $1.75 of machine time.

Disposable screens, by the minute5 templates and 3 snapshots a screenCreated and deleted through the APIUpdated 2026-11-23

The arithmetic first

A disposable screen is $0.035 an hour while it runs, charged by the minute, and $0.003 an hour stopped. Running a hundred screens for half an hour costs the same as one screen for fifty hours: $1.75. A thousand five-minute runs are $2.92. A thousand half-hour runs are $17.50. Parallelism changes how long you wait, not what you pay.

Those numbers are the machines only. The model tokens are yours and will dwarf them: on the public Cua Driver Bench data, one trial of one ten-task desktop benchmark cost about $0.40 to $0.60 in model spend with a mid-sized model. The machine is the cheap part of an eval, which is not what most people assume.

Make the machines identical

  • Build one screen the way the run needs it, with the app, the fixtures and the settings in place.
  • Save it as a template, and start every run's screen from that template. Each organization gets five templates, and each screen three snapshots.
  • Snapshot before a destructive step so a failed trial can be reset rather than rebuilt.
  • Keep the image's own defaults in mind: in our own tree eval, LibreOffice's autocorrect cost about three turns in every single run. An eval measures your harness as much as your agent.

Drive the fleet from your own code

The API creates, starts, stops and deletes screens; the hosted MCP server deliberately cannot, so an agent cannot spin up machines on your account. A benchmark runner is therefore a plain script with an API key: create from the template, run the trial, read the result back through the API rather than trusting the agent's report, delete the screen.

  • Write calls take an idempotency key, so a retry after a timeout does not run the trial twice.
  • Name the run with a header and the calls group under one label instead of a flat log.
  • Check the outcome on the machine, not in the transcript. Our own eval passed or failed each run on eight checks read through the API.
  • Each organization has a cap on how many screens can exist at once, so ask before you plan a thousand in parallel.

What we learned running one

Our tree eval was ten runs of one task, which is small, and it still produced a 19% spread between two runs of the identical setup. Single runs prove nothing. It also showed where the time really goes: about 60% model turns, about 40% tools, and most of the tool time was the agent's own fixed waits, 59 to 82 seconds a run. If you are optimising an eval, start there rather than with the machines.

And hold one thing fixed at a time. Most published computer-use leaderboards vary the model between rows, which is why they disagree with each other; the one dataset that holds the model steady reorders the drivers.

Questions people ask

What does it cost to run 1,000 computer-use trials?

In machine time, $2.92 if a trial takes five minutes and $17.50 if it takes thirty, on disposable screens billed by the minute. The model spend will be larger, on the order of a few hundred dollars at $0.40 a trial.

Can I run hundreds of screens at once?

Each organization has a screen cap so a runaway script cannot run up the bill, and we raise it on request. The price is the same whether the hours run side by side or one after another.

Can an agent create its own screens for a run?

Not through the MCP server, which has no tool for it. Use the API with a key you control, which is also where the idempotency keys and the run labels are.

Do you publish your own benchmark results?

We publish the measurements we have made, including the one that found six faults in our own tooling, and we name whose data it is when it is not ours. We do not have a Kubi arm on a public leaderboard, and we will say what it measured when we do.

Give your agent a desktop that stays on.

Private beta. Founding price $20 a screen a month, kept while you stay.

Get early access