Skip to content

Recipe: JavaScript-Rendered Pages

Goal

Read a page whose content appears only after JavaScript runs in the browser, such as a single-page app or a dashboard. Use the fast fetch_webpage first, and fall back to fetch_webpage_rendered only when the page turns out to be an empty shell.

Prerequisites

  • The web extra for fetch_webpage (add extract for main-content extraction).

  • The browser extra and Chromium for fetch_webpage_rendered:

    pip install "doc-processing-std-tools[web,extract,browser] @ git+https://github.com/rzoeb/doc-processing-std-tools.git@v0.6.1"
playwright install chromium            # in a container: playwright install --with-deps chromium

  • fetch_webpage_rendered belongs to the browser server, which is off unless enabled. You don't need the rest of the browser tools to use it (step 4).

Steps

Tool calls are written as Python calls; the model sends the same arguments as JSON.

1. Fetch the page normally

fetch_webpage(url="https://example.com/dashboard")

On a page built by JavaScript, the HTML the server sends holds little more than a title and an empty <div id="root">, and the result says so with likely_js_rendered:

{
  "title": "Results dashboard",
  "extractor": "stdlib",
  "text": "Results dashboard",
  "total_chars": 17,
  "likely_js_rendered": true
}

likely_js_rendered is true when the page has little text and several scripts, or an empty app root such as <div id="root"></div>. fetch_webpage takes milliseconds and runs no JavaScript, so it stays the first choice: most pages don't need a browser.

2. Fetch it again, rendered

fetch_webpage_rendered(url="https://example.com/dashboard")

Chromium loads the page, runs its JavaScript and returns the same JSON object as fetch_webpage, plus "rendered": true:

{
  "url": "https://example.com/dashboard",
  "final_url": "https://example.com/dashboard",
  "status": 200,
  "content_type": "text/html",
  "title": "Results dashboard",
  "format": "text",
  "extractor": "trafilatura",
  "text": "Quarterly results\nRevenue grew 12% this quarter.\nDetails",
  "truncated": false,
  "total_chars": 56,
  "offset": 0,
  "next_offset": null,
  "has_more": false,
  "byte_cap_reached": false,
  "rendered": true
}

It takes seconds rather than milliseconds. Each call uses a fresh browser context with no cookies or stored sign-ins, which is closed afterwards, so it reads pages as a signed-out visitor. For a page behind a sign-in, drive the browser tools instead (Portal Download to Text).

The arguments match fetch_webpage's:

fetch_webpage_rendered(url="https://example.com/dashboard", format="markdown", include_assets=True)
fetch_webpage_rendered(url="https://example.com/dashboard", offset=10000)   # the next part

# Quarterly results

Revenue grew 12% this quarter.

[Details](https://example.com/details)

With include_assets=True, the result also lists the page's links and images, with absolute URLs.

4. Give an agent both tools

fetch_webpage's description already tells the model to retry with fetch_webpage_rendered when likely_js_rendered is true and that tool is available. So attach the two tools, and nothing else from the browser server:

from doc_processing_toolkit import Agent
from doc_processing_std_tools import browser_server, web_search_server

agent = Agent(
    model="claude-sonnet-5",
    tools=[
        {"type": "fastmcp", "server": web_search_server, "allowed_tools": ["fetch_webpage"]},
        {"type": "fastmcp", "server": browser_server, "allowed_tools": ["fetch_webpage_rendered"]},
    ],
)

Through the composed launcher, set DPTTOOLS_ENABLE_BROWSER=true in the server's env. That adds all 19 browser tools (75 tools in all); use allowed_tools on the session to show the model only the ones it needs.

Internal sites

fetch_webpage_rendered uses the browser's network policy, which refuses private and internal addresses:

Error: Navigation refused: localhost resolves to a private or internal address; the browser may not reach it (add it to DPTTOOLS_BROWSER_ALLOW_HOSTS if it is a trusted internal host).

To read a trusted internal host, add its exact hostname to both allow-lists: DPTTOOLS_WEB_FETCH_ALLOW_HOSTS for fetch_webpage and DPTTOOLS_BROWSER_ALLOW_HOSTS for fetch_webpage_rendered. DPTTOOLS_BROWSER_ALLOWED_DOMAINS and DPTTOOLS_BROWSER_BLOCKED_DOMAINS apply to the rendered fetch as well.

What to validate

  • fetch_webpage on the shell returns likely_js_rendered: true, without the content.
  • fetch_webpage_rendered returns rendered: true, the same title, and the content in text.
  • format="markdown" keeps headings and links, and include_assets=True lists the links with absolute URLs.
  • Paging works the same as fetch_webpage's: offset, next_offset, has_more.
  • No browser session or tab is left open after a call.
  • A private address is refused unless its hostname is in DPTTOOLS_BROWSER_ALLOW_HOSTS, and file: URLs are refused.

Test sources

  • tests/integration/test_fetch_rendered.py: the shell and rendered fetches of the same page, on real Chromium (skipped when Chromium is missing).
  • tests/unit/test_fetch_webpage_structured.py (TestJsHint): when likely_js_rendered is set.
  • Web server: the result's fields, and the fetch's network policy.
  • Browser server: the rest of the browser tools, and its settings.