Skip to content

Web Server

web_search_server (doc_processing_std_tools.web_search, MCP name web-search-tools) gives an agent web search through a configured provider, a page fetcher and an image fetcher with SSRF protection, and a formatter that adds citation markers.

from doc_processing_std_tools import web_search_server
Extra Installs Needed for
web httpx, Pillow every tool here (Pillow for fetch_image)
extract trafilatura optional: main-content extraction and better Markdown in fetch_webpage

[all] includes both.

Tools

Tool Key parameters Returns
web_search query, num_results=5 (capped at 20), search_type="general" (or "news"), search_depth="basic" JSON {"provider", "query", "results": [{"title", "url", "snippet"}], "usage"}
fetch_webpage url, format="text" (or "markdown", "html"), max_length=10000, offset=0, include_assets=False, structured=True, extract_text=True A JSON object with the page's text and metadata (below); the plain text with structured=False
fetch_image url A caption (source, format, size) followed by the image for the model to look at
format_search_results results — the web_search output as it is, or a bare JSON array of {title, url, snippet or content} Numbered entries ([1] Title) followed by a References: list

Environment variables

Variable Default Effect here
DPTTOOLS_WEB_SEARCH_PROVIDER none tavily, serper or brave. Required by web_search
DPTTOOLS_WEB_SEARCH_API_KEY none The provider's API key. Required by web_search
DPTTOOLS_WEB_FETCH_TIMEOUT 30 HTTP timeout for fetch_webpage and fetch_image, in seconds, applied to each network operation; connecting is capped at 10 s
DPTTOOLS_WEB_FETCH_MAX_BYTES 2000000 Most bytes fetch_webpage downloads from one response; a longer page is cut and the result says byte_cap_reached
DPTTOOLS_WEB_FETCH_MAX_REDIRECTS 5 Most redirects the fetch tools follow
DPTTOOLS_WEB_FETCH_ALLOW_HOSTS empty Comma-separated exact hostnames the fetch tools may reach although they resolve to a private address
DPTTOOLS_FETCH_IMAGE_MAX_BYTES 10000000 Largest image fetch_image downloads
DPTTOOLS_IMAGE_MAX_EDGE_PX, DPTTOOLS_IMAGE_MAX_MB 1568, 3 The size fetch_image scales an image to before the model sees it (Images)

All except the first two are new in 0.6.0. fetch_webpage, fetch_image and format_search_results work without a search provider. The Configuration page lists every variable.

Upgrading from 0.5.0

  • fetch_webpage returns a JSON object by default. Code that used its result as the page text should read the text field, or pass structured=False for the plain string 0.5.0 returned.
  • Non-text responses are refused. Images, PDFs and other binary content now return an error (with their size) instead of garbled text; plain text and JSON are no longer passed through the HTML-to-text step.
  • fetch_webpage's timeout is DPTTOOLS_WEB_FETCH_TIMEOUT, no longer DPTTOOLS_SANDBOX_TIMEOUT. If you raised the sandbox timeout to allow slow pages, set the new variable instead.
  • web_search output has a usage key, and search_type="news" no longer forces Tavily's 2-credit advanced depth. Strict-schema parsers must allow the new key.

fetch_webpage

The result

{
  "url": "https://example.com/report",
  "final_url": "https://www.example.com/report",
  "status": 200,
  "content_type": "text/html",
  "title": "Annual report",
  "format": "text",
  "extractor": "trafilatura",
  "text": "...",
  "truncated": true,
  "total_chars": 48210,
  "offset": 0,
  "next_offset": 10000,
  "has_more": true,
  "byte_cap_reached": false,
  "likely_js_rendered": false
}
Field Meaning
final_url, status, content_type Where the request ended after redirects, and what the server sent
title The HTML <title>; null for non-HTML content
format How HTML was returned: text, markdown or html. text for non-HTML content
extractor trafilatura (the extract extra found main content), stdlib (the built-in extractor) or none (returned as-is)
text At most max_length characters, starting at offset
total_chars, has_more, next_offset The full length; when has_more is true, call again with offset=next_offset
truncated More text exists past this window, or the download was cut
byte_cap_reached The response was larger than DPTTOOLS_WEB_FETCH_MAX_BYTES and was cut, so the text ends early
likely_js_rendered The page looks built by JavaScript in the browser: little text with several scripts, or an empty app root such as <div id="root"></div>

With include_assets=True the object also has:

  • images — up to 30 {url, alt, width, height, source}, from <img>, og:image / twitter:image and icons;
  • links — up to 100 {url, text};
  • assets_truncated — true when a list hit its cap.

All URLs are absolute. Nothing is downloaded: pass an image URL to fetch_image to look at it.

What it reads, and how

  • Text content only: HTML, plain text, Markdown, JSON, XML, CSV, other text/* types, and +json / +xml types. Anything else is refused from its headers, before the body is downloaded. A response with no content type is downloaded and refused if it looks binary.
  • Only HTML is converted. An HTML page — including one served as text/plain — becomes text, Markdown or stays HTML, per format. JSON, plain text, CSV and other text are returned as they are.
  • Extraction. With the extract extra, text and markdown hold the page's main content, without navigation, sidebars and footers. Without it, or when trafilatura finds no main content, the built-in extractor converts the whole page; its Markdown keeps headings, lists, links and tables. structured=False always uses the built-in extractor.
  • Encoding comes from a byte-order mark, then the Content-Type charset, then an HTML <meta charset>, then UTF-8. Undecodable bytes are replaced rather than failing.
  • extract_text=False is the older spelling of format="html".

SSRF policy

fetch_webpage and fetch_image share one policy for reaching the network:

  1. Only http and https URLs are accepted.
  2. Each hop's hostname is resolved once, and every resolved address must be public. Refused: every block the IANA special-purpose address registries list as not globally reachable (private, loopback, link-local 169.254.0.0/16, which covers cloud metadata endpoints, carrier-grade NAT 100.64.0.0/10, 0.0.0.0/8, the IETF protocol block 192.0.0.0/24, unique-local fc00::/7, and the documentation and benchmarking ranges), multicast, reserved addresses, and any IPv6 address outside the public unicast block 2000::/3. The blocks are listed in the code, so the result is the same on every Python release. An IPv6 address that carries an IPv4 one (IPv4-mapped ::ffff:…, 6to4, Teredo, NAT64 64:ff9b::/96) is judged by that IPv4 address, so ::ffff:127.0.0.1 is refused like 127.0.0.1.
  3. The connection goes to the first validated address itself, with the original hostname in the Host header and for TLS, so a second DNS answer cannot redirect it (DNS rebinding).
  4. Redirects are followed by hand, at most DPTTOOLS_WEB_FETCH_MAX_REDIRECTS, and every hop is checked again.
  5. No credentials are sent — std-tools sets no auth headers — so a redirect cannot carry any to another host.
  6. Bodies stop at the byte limit, whatever the server sends.
  7. Network errors are reported without the URL or the resolved address.

An operator can let named internal hosts through with DPTTOOLS_WEB_FETCH_ALLOW_HOSTS — exact hostnames, no wildcards, empty by default. Those hosts are still resolved once and pinned; only the private-address check is skipped for them.

Upgrading from 0.6.0

The address check is wider in 0.6.1. Carrier-grade NAT addresses (100.64.0.0/10, which Tailscale uses), 0.0.0.0, the IETF protocol block 192.0.0.0/24 and IPv4 addresses inside IPv6 forms were reachable before and are now refused. If the fetch tools reach an internal host at such an address, add its hostname to DPTTOOLS_WEB_FETCH_ALLOW_HOSTS, and to DPTTOOLS_BROWSER_ALLOW_HOSTS for the browser.

fetch_image

fetch_image(url) downloads one image and shows it to the model:

  • PNG, JPEG, GIF and WebP only. The content type is checked before download, and the real format is then checked with Pillow, so a mislabelled file is refused.
  • An image whose declared size is over DPTTOOLS_FETCH_IMAGE_MAX_BYTES is refused before download; one that turns out larger while downloading is refused too. Images over 64 million pixels are refused.
  • The image is scaled to the model limits (1568 px, 3 MB by default) before it is returned; the caption says when it was scaled, for example Image from https://example.com/chart.png: PNG, 3200x1800 px, scaled to 1568x882 for the model.
  • The same SSRF policy as fetch_webpage applies, including DPTTOOLS_WEB_FETCH_ALLOW_HOSTS.

Search providers and usage

Provider Authentication search_type="news" search_depth usage in the result
Tavily Authorization: Bearer <key> header Searches the news topic basic (1 credit), advanced (2 credits), fast (1), ultra-fast (1) What Tavily reports, plus search_depth, request_id and response_time
Serper X-API-KEY header Searches news Ignored {"provider_reported": false}
Brave X-Subscription-Token header Passes a news search type Ignored {"provider_reported": false}
  • For Tavily, std-tools asks for usage and passes on what Tavily returns, marked "provider_reported": true. If Tavily reports nothing, the result carries an estimate instead, marked "estimated": true with estimated_credits.
  • Serper and Brave report no cost, and std-tools does not invent one.
  • search_type="news" does not change the depth: pass search_depth="advanced" explicitly to pay for it.
  • Provider calls use a fixed 30-second timeout. Provider errors are returned with request URLs replaced by <url> and the API key by <redacted>.

Examples

Tool calls are written as Python calls; the model sends the same arguments as JSON.

web_search(query="LibreOffice headless convert-to options", num_results=5)
web_search(query="central bank rate decision", search_type="news", search_depth="advanced")

fetch_webpage(url="https://example.com/article", format="markdown")
fetch_webpage(url="https://example.com/article", offset=10000)        # the next part
fetch_webpage(url="https://example.com/gallery", include_assets=True)
fetch_image(url="https://example.com/images/chart.png")

format_search_results(results=search_output)   # the whole web_search output

Pitfalls

  • Read long pages in parts. text holds max_length characters; follow next_offset while has_more is true rather than raising max_length a long way. An offset past the end is an error.
  • likely_js_rendered means the text is probably incomplete. fetch_webpage reads the HTML the server sends and runs no JavaScript.
  • Binary files are refused, not downloaded. For a PDF or an Office file, download it with execute_command and use the Documents tools; for an image, use fetch_image.
  • Internal hosts need the allow-list. Every non-public address is refused unless its hostname is in DPTTOOLS_WEB_FETCH_ALLOW_HOSTS.
  • An unknown format, search_type or search_depth is an error, not a silent default.
  • Search results are not truncated by DPTTOOLS_OUTPUT_TRUNCATION_LIMIT; num_results bounds them.
  • A cancelled fetch keeps running. Through the composed launcher, tools run on worker threads, which cannot be interrupted; a fetch ends when it completes or times out.

JavaScript-rendered pages

When likely_js_rendered is true, the text is probably incomplete. If the Browser server is enabled, retry with fetch_webpage_rendered: it returns the same JSON object (plus "rendered": true) from a page Chromium loaded with its JavaScript run, using a fresh, credential-free context behind the browser's network policy. It is slower — seconds rather than milliseconds — so fetch_webpage stays the first choice. fetch_webpage's description tells the model to do this when the tool is available.

Recipes