A safe web access tool for an AI agent has a narrow job, a validated input, and an output that says what was actually retrieved. Give discovery, single-page reading, and browser rendering separate descriptions so the agent can ask for the right operation. Enforce URL and budget rules in application code, then return source identity, content status, errors, and usage as structured fields. A good description helps routing; it cannot authorize a request, prove that a page was read completely, or make an untrusted page safe to obey.
Decide which web action the agent is requesting
An agent can know the question without knowing a source URL. In that case, Search should return candidates for the agent to select. Once it has a public URL, Fetch can read content already present in HTML. Render is a conditional second path when the expected content only appears after JavaScript runs. A screenshot records visible state; it does not substitute for extracted text. Login, clicking, form submission, and other state-changing interactions belong to a separately authorized browser workflow.
| Agent has | Tool to offer | A usable result | Stop or escalate when |
|---|---|---|---|
| A topic, but no URL | Search | Candidate URLs with titles and query context | A snippet is being treated as a full document |
| One public URL with useful HTML | Fetch | Markdown plus source and status fields | Expected passage or field is absent |
| A known URL whose target content requires JavaScript | Render | Rendered content with the same source checks | It needs login or interaction beyond page reading |
| A need to preserve visible appearance | Screenshot | Image artifact linked to a URL and capture context | A screenshot is being used as the only textual evidence |
The descriptions should tell the model when and when not to call each tool. “Browse any website and return an answer” leaves the operation, scope, and evidence requirement undefined. A better Fetch description is: “Read one permitted public URL whose useful content is in HTML. Return bounded Markdown and source status. Use Search when the URL is unknown; request Render only after a content check shows that Fetch missed JavaScript-loaded material. Do not use for login, clicks, forms, or access-control bypass.” The Page Search API's discovery contract gives an agent a candidate list; the Fetch API's request and response fields are the concrete next step for reading a selected URL.
Put the input boundary in code, not in the description
The tool interface should expose only decisions the agent is allowed to make. The application should keep domain policy, maximum result size, timeouts, credentials, and spending limits outside the model's arguments. Validate a URL after parsing it, reject unsupported schemes and private or disallowed destinations, and apply the same checks again after redirects. The agent must not be able to add arbitrary headers, set an unrestricted proxy, or switch a read operation into a write operation through extra JSON properties.
Here is an illustrative function definition for a known-URL read. It is an adapter schema, not an AnyCrawler API request. Search would be a separate function that accepts a query rather than a URL. In OpenAI's function-calling guide, strict mode constrains the shape of generated arguments; the executor still has to check that a URL is permitted and that the requested operation fits the user's authority.
{
"type": "function",
"name": "read_public_page",
"description": "Read one permitted public URL. Use only after a URL is known; do not use for search, login, clicks, forms, or bypassing access restrictions. Return source and failure fields with bounded content.",
"strict": true,
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Absolute HTTPS URL of one public page selected for reading."
},
"expected_content": {
"type": "string",
"description": "A short heading, phrase, or field that must appear for this task."
}
},
"required": ["url", "expected_content"],
"additionalProperties": false
}
}
The expected_content check is deliberately task-specific. A page can return HTTP 200 and still be an empty JavaScript shell, a cookie interstitial, or the wrong redirect target. Do not turn a missing expected phrase into an invented successful read. If you wrap the adapter as an MCP tool, the current MCP tool specification defines inputSchema, optional outputSchema, and a tool-result error path. Map the same policy into that interface without claiming that either schema format guarantees the model will choose correctly.
Return evidence that survives the model's next step
A useful output is a typed record, not just a Markdown blob. Keep the submitted URL, actual final URL when the provider exposes it, page status, content validation result, and a short content window. Include a truncation flag so the model cannot mistake a bounded excerpt for the whole page. A null field should mean “not returned or not verified,” not “probably the submitted URL.” The Fetch product documentation describes requested_url, final_url, canonical_url, status_code, results.markdown, credits_used, and error fields; an adapter should preserve those distinctions when available.
| Field in the adapter | Why keep it | If unavailable |
|---|---|---|
requested_url, final_url, canonical_url |
Distinguish the request, redirect destination, and declared publication identity | Keep an explicit null; do not silently equate them |
source_status, content_valid |
Separate network success from task-level completeness | Return an incomplete/error state |
markdown, truncated |
Bound model context while showing whether text was cut | Return empty content with an explanation |
error_code, retryable |
Let the orchestrator decide whether another attempt makes sense | Keep unknown retryability explicit |
credits_used, request ID, capture time |
Audit cost and support a later source check | Preserve null rather than estimating |
Source text must be treated as data. A page may contain instructions aimed at the agent; those words do not gain authority by arriving through a web tool. OWASP's prompt-injection guidance describes this indirect path. Keep the source boundary visible in the result and require separate application approval for any privileged follow-up action. Content filtering can help, but it is not a proof that arbitrary pages are safe.
Test the envelope on a public page
This small Node.js example uses AnyCrawler's public, no-key free crawl endpoint with https://example.com/, a documentation example domain. It exercises a real read and normalizes the fields the free response actually supplies. The free endpoint is a one-page demonstration, not the authenticated POST /v1/crawl/page Fetch integration described above. For production, replace the transport with your authenticated call, retain the same validation and output checks, and handle its documented error fields.
const target = new URL("https://example.com/");
if (target.protocol !== "https:" || target.hostname !== "example.com") {
throw new Error("URL outside this example's allowlist");
}
const endpoint = new URL("https://api.anycrawler.com/free/v1/crawl");
endpoint.searchParams.set("url", target.href);
const response = await fetch(endpoint, { signal: AbortSignal.timeout(20000) });
const raw = await response.json();
const markdown = typeof raw.results?.markdown === "string" ? raw.results.markdown : "";
const contentValid = markdown.includes("Example Domain");
const result = {
ok: response.ok && raw.ok === true && raw.status_code === 200 && contentValid,
requested_url: raw.requested_url ?? target.href,
final_url: raw.final_url ?? null,
canonical_url: raw.canonical_url ?? null,
source_status: raw.status_code ?? null,
content_valid: contentValid,
markdown: markdown.slice(0, 1200),
truncated: markdown.length > 1200,
credits_used: raw.credits_used ?? null,
error_code: raw.error_code ?? null,
retryable: raw.retryable ?? null
};
console.log(JSON.stringify(result, null, 2));
In the recorded public probe, HTTP and source status were 200, ok was true, and Markdown began with # Example Domain. The response did not include final_url, canonical_url, or credits_used, so the normalized record leaves them null. That is a useful test of the adapter's honesty, not evidence that the authenticated endpoint omits those fields. The example's domain allowlist is intentionally fixed; a real service needs its own policy and redirect checks.
Make failure states actionable
Before passing text to the model, ask whether the requested page, returned page, and expected content still describe the same source. A successful HTTP response with missing task content should become content_invalid, not a citation. A redirect outside the approved destination set should stop the read. A timeout or rate limit can be retried only within the application's budget and the provider's error semantics; an authentication or permission failure needs a different fix. If one source fails in a multi-source job, keep its error record beside the successful sources instead of erasing it.
The model can then make a bounded next request: choose another search candidate, ask for Render when the content check suggests a JavaScript shell, or report that the evidence remains incomplete. Preserve the provider's request ID when available so a human can diagnose a bad result. Keep a separate capture time from a page's publication date, and never infer freshness from a current tool call if a cache may have supplied older content.
Which boundary should remain a separate tool?
A combined read interface can simplify a small application, but it can also hide whether a call searched, fetched one URL, rendered JavaScript, or performed an action. The decision is whether callers can still see distinct cost, latency, permission, and evidence states for each branch. If they cannot, expose separate tools and let the orchestrator select a branch after validating the previous result. Start by wiring one permitted page read to the Fetch API contract, then test empty content, redirects, and error records before expanding the agent's web access. What additional operation would require a new permission boundary in your application?
Frequently asked questions
Should one web tool handle both search and page reading?
It can, if the application still exposes the chosen operation and validates its distinct inputs and outputs. A search query returns candidate URLs and snippets; a page read starts from one selected URL and returns content evidence. If a single tool hides that difference, agents and operators can mistake a search snippet for a verified source. Separate tools usually make the boundary clearer, especially when permissions, costs, and failure handling differ.
Can a strict JSON schema make an agent's web access safe?
Strict schemas help constrain the arguments a model emits. They do not decide whether the user authorized a domain, whether a redirect is acceptable, whether the returned page contains the requested evidence, or whether page text is trying to issue instructions. The executor must enforce those checks after it receives the arguments and again when it receives the provider's response. Treat the schema as an interface contract, with policy and authorization implemented separately.
When should the agent escalate from Fetch to Render?
Escalate only after a task-level content check indicates that the useful page material is missing from the HTML read and JavaScript execution is a plausible reason. A 200 response by itself does not show that the expected heading, table, or claim was extracted. Recheck the same expected content after Render and record which path produced it. If the page needs a login, click, or form submission, route that work to a separately authorized browser interaction instead.
What if the provider does not return a final URL or credit count?
Keep those fields null and state what was observed. The submitted URL is not proof of the final redirect destination, and an assumed credit count is not a usage record. You can still use the content for a bounded task if status and the expected passage are verified, but avoid making identity or cost claims that require the missing fields. If the missing field is essential to citation or accounting, treat the result as incomplete and use a path that provides it.
How much page content should the tool return to the model?
Return only enough content to satisfy a defined task, along with a truncation indicator and source identity. A fixed cap keeps one large page from consuming the whole context, but the cap must not turn a partial excerpt into a claim about the entire document. If the required passage falls outside the first window, use a follow-up extraction or section selection step. Store the complete evidence separately when your workflow requires later review.
