Watching an agent, and taking the screen back
Sooner or later the agent reaches a step a person has to do: a password, a CAPTCHA, a judgement call. The usual answer is a VNC stream both parties type into at once, which the measurements say is a bad idea. A screen uses a lock instead: one party holds the desktop, the other gets a clear error and waits.
Why a lock and not a shared mouse
The public Cua Driver Bench data has a background condition: the same desktop tasks, with a person typing in front of the agent. On Linux the pass rate falls from about two thirds in the foreground to 12.5%, and one driver on Windows scores 0 of 240. One desktop has one keyboard, and two parties using it at once is a race neither wins.
So the question is not how to share a mouse gracefully. It is who holds it, and how the other one is told.
What happens when you take control
- Press Take control in the dashboard and confirm, and your keyboard and mouse go to the desktop.
- While you hold it, the agent's screenshots, actions, commands, file calls, stop, restart and delete all get 409 CONTROL_HELD. It cannot look while you type, and nothing it planned lands on top of your work.
- It can still read status and history, and commands already running keep running: Kubi refuses calls, it does not freeze processes.
- Control ends when you press Release, or after five minutes with no input from you, with a warning in the last minute. A key or a mouse move resets the timer, and closing the tab just lets it run out.
- After you have typed, an action the agent planned on an older screenshot gets 409 STALE_OBSERVATION, so it looks again instead of clicking blind over your changes.
The last one is the quiet one that matters. An agent acting on a stale picture of the screen is how automation does real damage, and the screenshot id in every action is what makes that refusable rather than likely.
Showing the screen to someone who should only watch
An agent can hand a person a view-only link to its live screen with one API call. The link works once, has to be opened within 60 seconds, and ends after an hour. It is the right thing to send in a chat message that says "this is what I am looking at", and it gives away no control and no key.
Kubi carries no messages between your agent and you: when it needs a person it asks the way it always does, in its own chat or notifications, and the link is what it attaches.
What this gives you that a stream does not
- An honest answer to "what did it do?": the history of calls with their screenshots, and your own time on the screen marked as taken and released rather than recorded keystroke by keystroke.
- A safe place to type a password: nothing you do in the live view is written to the screen's history.
- A rule the agent can program against. 409 CONTROL_HELD is a documented state, not a stream that silently fights back.
- No lock between agents, deliberately: any key with the permission can call a screen, so one agent per screen is the shape to use.
Questions people ask
Can a person and an agent use one desktop at the same time?
What happens if I forget to release the screen?
Does taking control stop an agent running inside the screen?
Can I let a customer watch without giving them access?
Read next
The docs hold the reference; these pages hold what we measured.










