Skip to content

Recipe: Search, Then Summarise

Goal

Search the web, read the best sources in full, and write a summary that cites them, with the Web server's web_search, fetch_webpage and format_search_results.

Prerequisites

  • The web extra, and extract for main-content extraction (both are in all):

    pip install "doc-processing-std-tools[web,extract] @ git+https://github.com/rzoeb/doc-processing-std-tools.git@v0.6.1"

  • A search provider account (Tavily, Serper or Brave) and its API key.

Steps

Tool calls are written as Python calls; the model sends the same arguments as JSON. The results are from a live run, shortened.

1. Configure the provider

DPTTOOLS_WEB_SEARCH_PROVIDER=tavily        # or serper, brave
DPTTOOLS_WEB_SEARCH_API_KEY=tvly-xxx

Both go in the environment of the process that runs the tools: your own process in-process, or the server's env out of process. fetch_webpage and format_search_results need no provider.

web_search(query="LibreOffice headless convert-to pdf command line", num_results=5)

{
  "provider": "tavily",
  "query": "LibreOffice headless convert-to pdf command line",
  "results": [
    {"title": "Tip of the day: batch PDF conversion with LibreOffice - anarcat",
     "url": "https://anarc.at/blog/2019-10-08-libreoffice-pdf-batch",
     "snippet": "> Fear not, shell scripting and headless mode flies in to the rescue! ..."},
    {"title": "Convert-to command line parameter",
     "url": "https://ask.libreoffice.org/t/convert-to-command-line-parameter/840",
     "snippet": "More importantly, headless conversion silently does nothing if any LO is already open; ..."}
  ],
  "usage": {"provider_reported": true, "search_depth": "basic", "credits": 1,
            "response_time": 1.1, "request_id": "..."}
}
  • num_results defaults to 5 and is capped at 20.
  • search_type="news" searches news, for recent events.
  • With Tavily, search_depth="basic" costs 1 credit and "advanced" 2. usage reports what Tavily charged. Serper and Brave report no cost, and std-tools doesn't invent one.

3. Read the best sources

A snippet is a fragment, not a source. Pick the two or three results that answer the question, and read each one:

fetch_webpage(url="https://anarc.at/blog/2019-10-08-libreoffice-pdf-batch", format="markdown")

{
  "final_url": "https://anarc.at/blog/2019-10-08-libreoffice-pdf-batch/",
  "status": 200,
  "title": "Tip of the day: batch PDF conversion with LibreOffice - anarcat",
  "extractor": "trafilatura",
  "text": "# Tip of the day: batch PDF conversion with LibreOffice\n\nSomeone asked me today ...",
  "total_chars": 1560,
  "has_more": false,
  "likely_js_rendered": false
}
  • Long pages come in parts. When has_more is true, call again with offset=next_offset rather than raising max_length.
  • likely_js_rendered: true means the text is probably incomplete. Retry with fetch_webpage_rendered if the browser server is attached (JavaScript-Rendered Pages).
  • A PDF or Office file is refused, with its size. Download it with execute_command and read it with the Documents tools.
  • Private and internal addresses are refused, and so is a redirect to one.

4. Number the sources and cite them

format_search_results turns results into numbered entries and a reference list. Pass it the pages the model actually read, as a JSON array of {title, url, snippet}, so that the numbers match the sources used. Here the snippets are the points the summary takes from each page:

format_search_results(results=json.dumps([
    {"title": "Tip of the day: batch PDF conversion with LibreOffice - anarcat",
     "url": "https://anarc.at/blog/2019-10-08-libreoffice-pdf-batch",
     "snippet": "libreoffice --headless --convert-to pdf *.docx converts every DOCX file in the folder."},
    {"title": "Convert-to command line parameter",
     "url": "https://ask.libreoffice.org/t/convert-to-command-line-parameter/840",
     "snippet": "Headless conversion silently does nothing if LibreOffice is already open."},
]))

[1] Tip of the day: batch PDF conversion with LibreOffice - anarcat
libreoffice --headless --convert-to pdf *.docx converts every DOCX file in the folder.

[2] Convert-to command line parameter
Headless conversion silently does nothing if LibreOffice is already open.

---
References:
[1] Tip of the day: batch PDF conversion with LibreOffice - anarcat — https://anarc.at/blog/2019-10-08-libreoffice-pdf-batch
[2] Convert-to command line parameter — https://ask.libreoffice.org/t/convert-to-command-line-parameter/840

Entries are numbered in the order given, with the titles given. The model writes its summary with [1], [2] markers and ends with the reference list. The whole web_search output, passed as it is, works too: the numbers then follow the search results, with their snippets.

5. Give an agent the tools

from doc_processing_toolkit import Agent
from doc_processing_std_tools import web_search_server

agent = Agent(
    model="claude-sonnet-5",
    instructions=(
        "Answer with a short summary that cites its sources. Search with web_search, read the two "
        "or three best results with fetch_webpage, and use only what you read. Number the sources "
        "with format_search_results, cite them as [n] in the text, and end with the reference list."
    ),
    tools=[{"type": "fastmcp", "server": web_search_server,
            "allowed_tools": ["web_search", "fetch_webpage", "format_search_results"]}],
)
result = agent.run_sync("How do I convert a folder of DOCX files to PDF with LibreOffice?")

fetch_image is left out here because a summary doesn't need it. Add it when the model should look at a chart or photo from a page: fetch_webpage(..., include_assets=True) lists a page's image URLs.

What to validate

  • web_search returns provider, query, results (each with title, url and snippet) and usage, and Tavily's usage shows the credits charged.
  • Without the provider settings, web_search returns an Error: naming them, and the other three tools still work.
  • fetch_webpage returns the page's main text, and paging with offset returns the rest.
  • format_search_results numbers the entries from 1, and its reference list matches them.
  • An unknown search_type or search_depth is an error, not a silent default.
  • Error messages never contain the API key or the provider's request URL.

Test sources

  • tests/unit/test_web_search.py: each provider (TestWebSearch), usage reporting (TestSearchUsage), citation formatting (TestFormatSearchResults), argument checks (TestWebSearchInputValidation) and the network policy (TestSSRFProtection). HTTP is mocked, so no key is needed.
  • tests/unit/test_fetch_webpage_structured.py: the fetch_webpage result, paging and the content-type check.
  • tests/integration/test_web_search_mcp.py: the tools through the server.
  • Web server: every argument, the providers and the network policy.
  • Configuration: the fetch limits (DPTTOOLS_WEB_FETCH_*).