Computer API¶
Tools, not just functions
The computer_* functions below are registered as MCP tools by make_computer_server(). Their
exact model-visible descriptions are in the Tool Reference, and
how to use them is on the Computer (Desktop) server page.
Computer Server¶
doc_processing_std_tools.computer
¶
Computer (desktop) server: operate a whole desktop by screenshot, mouse and keyboard.
Modelled on Anthropic's computer_toolset_20260801: each of its 17
members is one MCP tool named computer_<member>. Coordinates are in the
pixels of the latest computer_screenshot. For anything inside web
pages, the browser server is the better tool: it reads the page itself.
Backends (DPTTOOLS_COMPUTER_BACKEND):
virtual(default): the agent's own Xvfb display per MCP session, in a Linux container. Start applications on it withexecute_command(DISPLAY=:<n> app &); every result names the display.host: the real screen, mouse and keyboard of the machine the server runs on, one session at a time, with a kill switch (top-left corner orDPTTOOLS_COMPUTER_KILL_HOTKEY) and a pause when the person at the machine uses it. Only with the user's knowledge: their screen goes to the model provider.
Off by default: the composed launcher includes this server only when
DPTTOOLS_ENABLE_COMPUTER=true. Optional packages are imported only when
a backend starts.
close_all()
¶
Stop every virtual display and the host listeners (also at exit).
computer_cursor_position(intent=None, ctx=None)
async
¶
Report where the mouse cursor is, in screenshot pixels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_double_click(x, y, modifiers=None, intent=None, ctx=None)
async
¶
Double-click at a point (for example to open a file or select a word).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
int
|
Horizontal position in screenshot pixels. |
required |
y
|
int
|
Vertical position in screenshot pixels. |
required |
modifiers
|
str | None
|
Keys to hold while clicking, such as "ctrl" or "shift+ctrl": key names joined by "+". |
None
|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_hold_key(text, duration, intent=None, ctx=None)
async
¶
Hold a key or chord down for a number of seconds (at most 30).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The key or chord, such as "shift" or "Right": single characters or key names joined by "+". |
required |
duration
|
float
|
Seconds to hold, at most 30. |
required |
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_key(text, repeat=1, intent=None, ctx=None)
async
¶
Press a key, a chord, or a space-separated sequence of them.
Each key is a single character or an xdotool key name made of letters, digits and "_", spelled as xdotool spells it: "Return", "Tab", "Page_Down", "super", "XF86AudioMute"; never a key code such as "119". Join keys into a chord with "+" ("ctrl+a", "alt+Tab") and separate a sequence with spaces ("ctrl+a BackSpace"). The plus key itself is "+" ("ctrl++" zooms in). Anything else is refused.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The key, chord or sequence. |
required |
repeat
|
int
|
Times to press the whole sequence, 1-100 (default 1). |
1
|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_left_click(x, y, modifiers=None, intent=None, ctx=None)
async
¶
Click the left mouse button at a point.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
int
|
Horizontal position in screenshot pixels. |
required |
y
|
int
|
Vertical position in screenshot pixels. |
required |
modifiers
|
str | None
|
Keys to hold while clicking, such as "ctrl" or "shift+ctrl": key names joined by "+". |
None
|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_left_click_drag(from_x, from_y, x, y, intent=None, ctx=None)
async
¶
Drag with the left mouse button from one point to another.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
from_x
|
int
|
Start, horizontal, in screenshot pixels. |
required |
from_y
|
int
|
Start, vertical, in screenshot pixels. |
required |
x
|
int
|
End, horizontal, in screenshot pixels. |
required |
y
|
int
|
End, vertical, in screenshot pixels. |
required |
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_left_mouse_down(intent=None, ctx=None)
async
¶
Press and hold the left mouse button where the mouse is.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_left_mouse_up(intent=None, ctx=None)
async
¶
Release the left mouse button where the mouse is.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_middle_click(x, y, modifiers=None, intent=None, ctx=None)
async
¶
Click the middle mouse button at a point.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
int
|
Horizontal position in screenshot pixels. |
required |
y
|
int
|
Vertical position in screenshot pixels. |
required |
modifiers
|
str | None
|
Keys to hold while clicking, such as "ctrl" or "shift+ctrl": key names joined by "+". |
None
|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_mouse_move(x, y, intent=None, ctx=None)
async
¶
Move the mouse to a point without clicking.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
int
|
Horizontal position in screenshot pixels. |
required |
y
|
int
|
Vertical position in screenshot pixels. |
required |
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_right_click(x, y, modifiers=None, intent=None, ctx=None)
async
¶
Click the right mouse button at a point (opens a context menu).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
int
|
Horizontal position in screenshot pixels. |
required |
y
|
int
|
Vertical position in screenshot pixels. |
required |
modifiers
|
str | None
|
Keys to hold while clicking, such as "ctrl" or "shift+ctrl": key names joined by "+". |
None
|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_screenshot(intent=None, ctx=None)
async
¶
Take a screenshot of the desktop.
x/y arguments of the other computer tools are in this screenshot's pixels. Take one before acting, and again to check a result.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_scroll(scroll_direction, scroll_amount=3, x=None, y=None, intent=None, ctx=None)
async
¶
Scroll at a point (default: the centre of the screen).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scroll_direction
|
Literal['up', 'down', 'left', 'right']
|
up, down, left or right. |
required |
scroll_amount
|
int
|
1-10 wheel steps (default 3). |
3
|
x
|
int | None
|
Horizontal position in screenshot pixels (with y). |
None
|
y
|
int | None
|
Vertical position in screenshot pixels (with x). |
None
|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_triple_click(x, y, modifiers=None, intent=None, ctx=None)
async
¶
Triple-click at a point (for example to select a line).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
int
|
Horizontal position in screenshot pixels. |
required |
y
|
int
|
Vertical position in screenshot pixels. |
required |
modifiers
|
str | None
|
Keys to hold while clicking, such as "ctrl" or "shift+ctrl": key names joined by "+". |
None
|
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_type(text, intent=None, ctx=None)
async
¶
Type text, as if on the keyboard, into whatever has focus.
Text only: tabs and newlines are typed, other control characters are refused. Press keys such as Delete, Escape or BackSpace with computer_key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The text to type. |
required |
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_wait(duration, intent=None, ctx=None)
async
¶
Wait a number of seconds (at most 30), for an application to respond.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
duration
|
float
|
Seconds to wait, at most 30. |
required |
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
computer_zoom(x0, y0, x1, y1, intent=None, ctx=None)
async
¶
Capture one region of the screen, enlarged, to read small text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x0
|
int
|
Left edge of the region, in screenshot pixels. |
required |
y0
|
int
|
Top edge of the region, in screenshot pixels. |
required |
x1
|
int
|
Right edge of the region, in screenshot pixels. |
required |
y1
|
int
|
Bottom edge of the region, in screenshot pixels. |
required |
intent
|
str | None
|
Optional note on why you are doing this (logged only). |
None
|
make_computer_server()
¶
Build the computer server with its 17 computer_<member> tools.
Backends and Key Chords¶
doc_processing_std_tools._computer_backends
¶
Backends for the computer server.
VirtualBackend: the agent's own X display per session. Xvfb plus a light window manager; input throughxdotoolwithDISPLAY=:<n>(the approach of Anthropic's computer-use reference container); capture throughmssbound to that display. Linux only.HostBackend: the real screen, mouse and keyboard of the machine the server runs on, throughpynputandmss, with a kill switch and a pause when the person at the machine uses it. Windows and Linux X11; macOS untested; Wayland unsupported.
Optional packages (mss, pynput, Pillow) are imported only when a
backend starts, so the computer server imports without the extra.
ComputerError
¶
Bases: Exception
An actionable failure; the tool returns "Error: <message>".
HostBackend
¶
The real desktop. One per process: there is only one real cursor.
TookOver
¶
Bases: ComputerError
Real input arrived while the agent was acting; the action stopped.
VirtualBackend
¶
One Xvfb display, its window manager and (optionally) a VNC view.
check_typed_text(text)
¶
Refuse control characters other than tab, newline and carriage return.
xdotool types a character with the key that produces it, and U+007F is
what the Delete key produces (U+001B is Escape's, U+0008 BackSpace's):
typed, it would press a key the confirmation gate never sees. Keys go
through computer_key.
chord_tokens(chord)
¶
"Ctrl+Shift+T" -> ["ctrl", "shift", "T"].
Built-in names and their synonyms become canonical lowercase names
(Control -> ctrl, Page_Down -> next). Any other name is kept
as written: X key names are case-sensitive, and xdotool ignores
iso_left_tab and reads 3270_enter as a key code.
Every token must be a single character or a key name; anything else (a space, a path, an option, a word xdotool reads as a command) is refused. A "+" where a key is expected is the plus key ("+", "ctrl++"); a chord that starts or ends with a separator, or has two keys without one between them ("ctrl+", "ctrl++a"), is refused rather than guessed at.
is_dangerous(chord)
¶
Whether the chord contains one of DANGEROUS_CHORDS.
Containment, not equality: extra keys ("alt+shift+F4", "ctrl+shift+w") do not make a chord safe. Side-specific X names and numeric spellings count as the key they name ("Alt_L" and "0xffe9" are alt, "U0077" is w, "KP_Delete" and "0xffff" are delete).