Skip to content

Computer (Desktop) Server

computer_server (doc_processing_std_tools.computer, MCP name computer-tools) lets an agent operate a whole desktop by screenshot, mouse and keyboard. It is modelled on Anthropic's computer use toolset (computer_toolset_20260801): each of its 17 members is one tool here, named computer_<member>, and the tools work as ordinary function tools with every model provider. New in 0.6.0.

Use it for native applications, operating-system dialogs and flows that cross applications. For anything inside web pages, the Browser server is the better tool, because it reads the page itself instead of looking at pixels.

Off by default — and host mode is the user's real computer

The composed launcher adds this server only when DPTTOOLS_ENABLE_COMPUTER=true. The default virtual backend gives the agent its own desktop inside a Linux container. The host backend operates the real screen, mouse and keyboard of the machine std-tools runs on: run it only with the user's knowledge, because every screenshot, with whatever is on the screen, goes to the model provider. Read Security before enabling it.

Install and enable

pip install "doc-processing-std-tools[computer] @ git+https://github.com/rzoeb/doc-processing-std-tools.git@v0.6.1"
# virtual backend (Linux): apt-get install xvfb xdotool openbox   (x11vnc optional)

The computer extra installs mss, Pillow and, except on Linux, pynput. On Linux the extra leaves pynput out, because its evdev dependency compiles from source and would break [all] on slim images. Linux host mode installs it separately:

pip install python-xlib && pip install --no-deps pynput
Route How
Composed launcher Set DPTTOOLS_ENABLE_COMPUTER=true
In-process Pass computer_server to Agent(tools=[...])

Optional packages are imported only when a backend starts, so the module imports without the extra.

How an agent uses the desktop

  • Look, then act. computer_screenshot returns an image of the desktop. The other tools take x and y in the pixels of the latest screenshot.
  • Large screens are scaled. A screen larger than DPTTOOLS_IMAGE_MAX_EDGE_PX (1568 px) is scaled down for the model, and the coordinates the model sends are mapped back to real pixels (Images). The default virtual display, 1440 × 900, is not scaled.
  • Every result names the display or monitor it acted on. Every tool takes an optional intent (logged only).
  • Starting applications on a virtual desktop: use execute_command with the display the results name, for example DISPLAY=:99 libreoffice --calc report.xlsx &.

Tools

Tool Key parameters What it does
computer_screenshot — A screenshot of the desktop (an image)
computer_zoom x0, y0, x1, y1 A region of the screen, enlarged up to 4×, to read small text
computer_left_click x, y, modifiers Click, optionally holding keys (ctrl, shift+ctrl)
computer_right_click x, y, modifiers Right-click
computer_middle_click x, y, modifiers Middle-click
computer_double_click x, y, modifiers Double-click
computer_triple_click x, y, modifiers Triple-click
computer_left_click_drag from_x, from_y, x, y Drag from one point to another
computer_mouse_move x, y Move the mouse without clicking
computer_left_mouse_down — Press and hold the left button where the mouse is
computer_left_mouse_up — Release the left button
computer_cursor_position — Where the cursor is, in screenshot pixels
computer_scroll scroll_direction, scroll_amount (1–10, default 3), optional x, y Scroll at a point, or at the screen centre
computer_type text Type text into whatever has focus. Tabs and newlines are typed; other control characters are refused (press keys with computer_key), new in 0.6.1
computer_key text, repeat=1 Press a key, a chord or a sequence, with xdotool key names: Return, ctrl+a, alt+Tab, ctrl+a BackSpace
computer_hold_key text, duration Hold a key or chord for up to 30 seconds
computer_wait duration Wait up to 30 seconds

With the computer server enabled, the composed launcher has 73 tools (56 + 17), or 92 with the browser server's core profile as well.

Backends

DPTTOOLS_COMPUTER_BACKEND chooses whose desktop the agent uses.

virtual (default): the agent's own desktop

  • Each MCP session gets its own Xvfb display (DPTTOOLS_COMPUTER_DISPLAY_SIZE, default 1440x900) with a light window manager. Input goes through xdotool on that display only, so sessions never see each other's desktops.
  • Linux only, so in practice a container — see the Deployment pattern.
  • At most DPTTOOLS_COMPUTER_MAX_SESSIONS (2) desktops are open at once; one unused for 10 minutes is closed.
  • DPTTOOLS_COMPUTER_VNC_PORT serves each display over VNC on 127.0.0.1 (the first session on that port, the next on port + 1; needs x11vnc), so a person can watch or take over.
  • The desktop sees the server's environment. Xvfb, the window manager, x11vnc and xdotool start with the server's environment variables, API keys included, and so does any program started from the desktop (the window manager's menu, for example). Give the computer server only the keys it needs: run it in a launcher process of its own, with no API keys in its environment.

host: the real screen

  • Operates the real screen, mouse and keyboard of the machine std-tools runs on, through pynput and mss.
  • Platforms: Windows and Linux X11. macOS is untested (it needs Accessibility and Screen Recording permission). Wayland is not supported.
  • One session at a time — there is only one real cursor. A second client is refused until the first has been idle for a minute.
  • DPTTOOLS_COMPUTER_MONITOR picks the monitor (default: the primary one); coordinates are relative to it, never to the combined virtual screen.
  • On Windows the server declares per-monitor DPI awareness, so screenshots and clicks share physical pixels.

Host-mode safeguards

These are always on in host mode, whatever the confirmation gate says:

  • Kill switch. Moving the mouse into the top-left corner of the screen, or pressing DPTTOOLS_COMPUTER_KILL_HOTKEY (default ctrl+alt+shift+k), stops the agent. Every later call returns an error until the server restarts.
  • Pause on human activity. If the user moved the mouse or typed in the last second, the next action is refused ("the user is using the computer right now"). On Windows, input that arrives during an action also stops that action ("the user took over").
  • On Linux X11 the system does not mark the agent's own input reliably, so std-tools ignores input during an action and for 0.4 s after it. There the pause, and the top-left corner, work between actions only. The kill hotkey works at any time, on both platforms.
  • Every screenshot goes to the model provider, with whatever is on the screen — other windows, notifications, messages. Close what the agent should not see.

The confirmation gate

DPTTOOLS_COMPUTER_CONFIRM takes the same three values as the browser's gate:

Value Effect
off (default) std-tools asks nothing; the client application approves actions through its own hooks
deny Key chords that close, quit or delete are refused: alt+f4, ctrl+w, ctrl+q, super+w, super+q, delete, shift+delete. A chord that contains one of them counts too (ctrl+shift+w, ctrl+delete, Alt_L+F4, alt+0xffc1), and so do a click's modifiers (modifiers="alt+F4"), new in 0.6.1
elicit std-tools asks a human through MCP elicitation before the first action of each host session ("Allow the agent to control this computer for this session?") and before each of those chords; it refuses if nobody can answer. The toolkit has no elicitation support today

The gate knows only these chords: an application's own shortcuts, keys such as XF86Close, and typed text, such as a command in a terminal, pass it. Nor can it tell what a click will do — pixels carry no element names — so approval in your application matters more here than for the browser. See Using with the Toolkit.

Environment variables

Variable Default Effect
DPTTOOLS_ENABLE_COMPUTER false Adds the computer server to the composed launcher
DPTTOOLS_COMPUTER_BACKEND virtual virtual or host
DPTTOOLS_COMPUTER_DISPLAY_SIZE 1440x900 Size of each virtual display (virtual only)
DPTTOOLS_COMPUTER_MAX_SESSIONS 2 Virtual desktops open at once (host always allows one)
DPTTOOLS_COMPUTER_VNC_PORT off VNC on 127.0.0.1 for each virtual display (needs x11vnc)
DPTTOOLS_COMPUTER_MONITOR primary The monitor host uses: primary or a number
DPTTOOLS_COMPUTER_KILL_HOTKEY ctrl+alt+shift+k The host kill switch chord
DPTTOOLS_COMPUTER_CONFIRM off The confirmation gate: off, deny or elicit

The Configuration page gives accepted values and when each is read.

Examples

A typical sequence on a virtual desktop, written as Python calls (the model sends the same arguments as JSON):

execute_command(command="DISPLAY=:99 libreoffice --calc /workspace/report.xlsx &")
computer_wait(duration=3)
computer_screenshot()                          # look before acting
computer_left_click(x=412, y=230, intent="select cell B4")
computer_type(text="2026-09")
computer_key(text="Return")
computer_screenshot()                          # check the result

Pitfalls

  • Coordinates come from the latest screenshot. After the screen changes, take a new screenshot before clicking again.
  • Screenshots are the expensive part. Each one costs image tokens; zoom into a region with computer_zoom to read small text instead of raising the image limits.
  • virtual needs Linux with xvfb, xdotool and a window manager installed. On Windows or macOS, only host is available.
  • Linux host mode needs pynput installed by hand (see above); without it the backend cannot start.
  • Key names are xdotool's, such as Return, BackSpace, Page_Down and super. Each key in computer_key, computer_hold_key and the click tools' modifiers must be a single character or a key name (letters, digits and _, such as XF86AudioMute); anything else is refused with an Error: naming it (new in 0.6.1). A name that starts with a digit must be a hex keysym (0x1008ff13) or a 3270_* name, never a key code such as 119. Names other than the common ones (ctrl, Return, Tab, F1–F12 …) are passed on as written, and X key names are case-sensitive: ISO_Left_Tab, not iso_left_tab.
  • In host mode the agent can see everything on the screen. Close windows and notifications the agent should not read, and tell the user that their screen goes to the model provider.
  • A killed host session stays killed until the server restarts.
  • computer_scroll at (0, 0) scrolls at the centre of the screen, because strict function calling (OpenAI) sends 0 as a placeholder for unused optional arguments. To scroll at the very top-left, move a pixel in: (1, 1).

Recipes