Skip to content

Recipe: Desktop Actions with Hooks

Goal

Let an agent drive a desktop application through the computer server, while your application shows a person each consequential action, with a screenshot, and lets them approve or decline it through a toolkit hook.

Prerequisites

  • A Linux container with a virtual desktop and the applications the agent should use: xvfb, xdotool, a window manager such as openbox, and std-tools with the computer extra. The Deployment page has a Dockerfile.
  • Your application, with doc-processing-toolkit, running outside the container or inside it.
  • A model that reads images: the agent works from screenshots.

The default virtual backend gives each MCP session its own display inside the container, so the agent never touches your real screen. host mode, which drives the real screen, is covered at the end.

Steps

1. Start the server where the desktop is

Open a stdio session to the composed launcher inside the container, with the computer server enabled:

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import get_default_environment, stdio_client

params = StdioServerParameters(
    command="docker",
    args=["run", "-i", "--rm", "-e", "DPTTOOLS_ENABLE_COMPUTER=true", "my-agent-image",
          "python", "-m", "doc_processing_std_tools"],
    env=get_default_environment(),
)

docker run -i connects the container's standard input and output to the session. Pass std-tools settings with more -e flags; the container has none of your application's environment. If your application runs in the same container, start python -m doc_processing_std_tools directly instead.

2. Open the application on the desktop

The first computer tool call starts the session's display, and every result names it:

computer_screenshot()
# Screenshot of display :99 (1440x900).   (followed by the image)

Your application then starts the program on that display with execute_command, through the same session, before the agent starts. Run it in the background with its output redirected, so that the command returns at once:

execute_command(command="DISPLAY=:99 nohup libreoffice --calc /workspace/report.xlsx > /dev/null 2>&1 &")

Step 4 shows these two calls in code. Don't give execute_command to the agent: it runs any shell command in the container, and an approval step built on screenshots can't judge a command. If the agent must start programs itself, add execute_command to the hook's tools in step 3, and show the person args["command"] instead of a screenshot.

3. Write the approval hook

A hook sees every before_tool_call event and can decline the call, so the tool never runs. For the desktop, the tools that act are the clicks, drags, mouse-down, typing and keys. computer_screenshot, computer_zoom, computer_cursor_position and computer_wait only look or wait.

The hook sees coordinates, not a button's name: computer_left_click(x=595, y=492). To show the person what that means, it takes a screenshot of its own through the same session, so of the same display, and shows it with the coordinates and the model's stated intent:

from doc_processing_toolkit.agent import HookDirective

ACTS = {
    "computer_left_click", "computer_right_click", "computer_middle_click",
    "computer_double_click", "computer_triple_click", "computer_left_click_drag",
    "computer_left_mouse_down", "computer_type", "computer_key", "computer_hold_key",
}

def make_approval_hook(session):
    async def approve(event):
        if event.event_type != "before_tool_call" or event.metadata["tool_name"] not in ACTS:
            return None
        name, args = event.metadata["tool_name"], event.metadata["arguments"]
        shot = await session.call_tool("computer_screenshot", {})
        image = next(c for c in shot.content if c.type == "image")   # base64 PNG in image.data
        if await ask_the_user(name, args, image):
            return None
        return HookDirective(action="skip_tool", message="The user declined this action.")
    return approve

ask_the_user stands for your application's approval step: draw a marker at args["x"], args["y"] on the image, show args.get("intent"), and wait for an answer. The hook's screenshot goes to your application only, never to the model.

4. Run the agent

import re

from doc_processing_toolkit import Agent

async with stdio_client(params) as (read, write), ClientSession(read, write) as session:
    await session.initialize()

    # Your application opens the program (step 2); the agent never gets execute_command.
    shot = await session.call_tool("computer_screenshot", {})
    display = re.search(r"display (:\d+)", shot.content[0].text).group(1)
    await session.call_tool("execute_command", {"command":
        f"DISPLAY={display} nohup libreoffice --calc /workspace/report.xlsx > /dev/null 2>&1 &"})

    agent = Agent(
        model="claude-sonnet-5",
        instructions=(
            "You operate a Linux desktop with the computer tools. Take a screenshot before acting, "
            "and give every action an intent that says what it is for. If the user declines an "
            "action, take a new screenshot and decide again."
        ),
        tools=[{"session": session, "name": "std-tools",
                "allowed_tools": ["computer_screenshot", "computer_zoom", "computer_left_click",
                                  "computer_type", "computer_key", "computer_wait"]}],
        hooks=make_approval_hook(session),
    )
    result = await agent.run("report.xlsx is open on the screen. Put 2026-09 in cell B4.")

The session belongs to the event loop that opened it, so run the agent with await agent.run() inside the async with block, not with run_sync. allowed_tools keeps the model's tool list to what the job needs, and every tool on it that acts is in the hook's list: the launcher with the computer server has 73 tools.

In a test run against a dialog that asked "Delete all 1,204 customer records?" with Keep and Delete buttons, the hook was called once, with the model's click and intent:

computer_left_click {'x': 595, 'y': 492, 'intent': 'Click Keep'}
  • Declined: the click did not run. The model reported that it had not clicked, said the dialog was still open, and asked how to continue.
  • Approved: the click ran, and the dialog recorded Keep.

5. Add a backstop in the server

Your hook is the approval step. These settings limit what can happen if a call gets past it:

Setting Effect
DPTTOOLS_COMPUTER_CONFIRM=deny Refuses the key chords that close, quit or delete: alt+f4, ctrl+w, ctrl+q, super+w, super+q, delete, shift+delete, also with extra keys and as a click's modifiers. It cannot judge what a click does
DPTTOOLS_COMPUTER_VNC_PORT Serves each display over VNC on 127.0.0.1 in the container (needs x11vnc), so a person can watch or take over through a tunnel you control
DPTTOOLS_COMPUTER_MAX_SESSIONS How many desktops can be open at once (default 2)

DPTTOOLS_COMPUTER_CONFIRM=elicit would ask through MCP elicitation, which the toolkit does not support, so with the toolkit it refuses those chords.

On the user's real screen (host mode)

DPTTOOLS_COMPUTER_BACKEND=host drives the real screen, mouse and keyboard of the machine std-tools runs on, so it does not belong in a container. The same hook works, and matters more there:

  • Get the person's consent before the agent starts, and tell them that every screenshot, with whatever is on the screen, goes to the model provider.
  • The safeguards are always on: moving the mouse to the top-left corner, or pressing ctrl+alt+shift+k, stops the agent; and an action is refused while the person is using the mouse or keyboard.
  • Only one session at a time.

See Isolated vs Attached and Security.

What to validate

  • The first computer call names the display, and an application started on it appears in the next screenshot.
  • The hook runs for each acting tool, with its coordinates and intent, and not for screenshots or waits.
  • A declined call does not run: the screen is unchanged, and the model is told it was declined.
  • An approved click lands where the model aimed.
  • With DPTTOOLS_COMPUTER_CONFIRM=deny, computer_key(text="ctrl+w") is refused.
  • Two sessions get two displays, and neither sees the other's windows.

Test sources

  • tests/integration/test_computer_virtual.py: clicks, typing, keys, drags, zoom and one display per session, on a real Xvfb display (run it in the Linux image with tests/integration/run_in_docker.py).
  • tests/unit/test_computer_unit.py: the confirmation gate, coordinate scaling, the host-mode safeguards and the tool count, with the backends replaced by a recording fake.
  • The hook mechanics (before_tool_call, skip_tool) are tested in doc-processing-toolkit; see its Hooks and steering page.