Choose a web scraping API when your AI agent needs information from a known page; choose browser automation when completing the task requires a sequence of interactions or control over a session. JavaScript alone does not decide between them: an API can render a page and still return only extracted content. Define success before selecting the tool. A reading task ends with verified data and provenance. An action task needs an observed outcome, explicit authority, and a recovery rule if the final state is uncertain.
Compare the task contract before the implementation
An API describes how your application calls a service. Browser automation describes capabilities such as navigating, selecting controls, and observing page state. These categories overlap: a hosted service can expose browser automation through an API. The useful comparison is between a narrow extraction request and a controllable interaction workflow.
Consider a pricing page. Reading the advertised monthly price is an extraction task. Selecting annual billing to reveal another price adds an interaction requirement if there is no documented URL or supported request option for that state. Buying the plan introduces a consequential action with a different completion rule. All three tasks can involve the same website, but they should not share an unrestricted tool contract.
| Reader task | Suitable starting capability | Evidence required to finish | Boundary to check |
|---|---|---|---|
| Read a public article at a known URL | HTML extraction | Expected passage, page identity, status | A successful response can contain the wrong document |
| Read text that appears after JavaScript runs | Rendered extraction | Required content in the rendered result | Rendering alone may not expose content behind a control |
| Select an option and read the changed panel | Browser interaction, or a documented state-specific API | Selected state and resulting content | Do not assume a render request performs the selection |
| Read an account-specific dashboard | Supported authenticated API or authorized browser session | Correct account, access scope, expected data | Public-page extraction does not establish session support |
| Submit a form or change a setting | Authorized action API or browser automation | Explicit resulting state or receipt | A click completing is not proof of the intended change |
| Find relevant pages without knowing their URLs | Search or discovery | Candidate URLs and selection rationale | Discovery output still needs source reading |
A screenshot can supplement any row when appearance matters. It records a visible state; it does not create the interaction that produced that state or prove that a server accepted a change.
Rendering is still a reading operation when the output is content
AnyCrawler's rendered-page extraction contract accepts a URL, executes the browser path, and returns extracted content with status and provenance fields. It documents wait controls for reading JavaScript-dependent pages. That is useful when the required information appears during loading, without your application directing a sequence of clicks or form entries.
Do not infer a general browser session, arbitrary clicking, or login support from the word “render.” Those capabilities need their own documented inputs and state model. If the requested detail appears only after selecting a tab, ask whether the extraction service supports that exact state. If it does not, the task needs another supported interface or an interaction workflow.
The same distinction applies to readiness. Waiting for navigation or a loading event does not prove that a particular product value, document section, or account panel is present. Set a content condition that can fail: the required heading exists, the selected billing period is explicit, or every required field has a source passage. Missing content should remain missing in the result, rather than becoming an invitation for the model to fill the gap.
Test the read contract on a permitted page
The following Python example calls the public endpoint used by AnyCrawler's free URL-to-Markdown tool. It needs Python and the curl executable, and uses a documentation fixture rather than a commercial target. The subprocess receives a fixed argument list without a shell. It checks the API response, the target-page status, and expected content before creating a record for the agent.
import datetime
import hashlib
import json
import subprocess
from urllib.parse import urlencode
source = "https://example.com/"
endpoint = "https://api.anycrawler.com/free/v1/crawl?" + urlencode({"url": source})
reply = subprocess.run(
["curl", "--silent", "--show-error", "--max-time", "30",
"--write-out", "\n%{http_code}", endpoint],
check=True, capture_output=True, text=True, encoding="utf-8",
)
body, api_status = reply.stdout.rsplit("\n", 1)
if api_status != "200":
raise RuntimeError(f"API HTTP {api_status}")
data = json.loads(body)
markdown = data.get("results", {}).get("markdown")
if data.get("ok") is not True or data.get("status_code") != 200:
raise RuntimeError("Target page did not pass the read contract")
if not isinstance(markdown, str) or "Example Domain" not in markdown:
raise RuntimeError("Expected page content is missing")
record = {
"requested_url": data.get("requested_url"),
"final_url": data.get("final_url"),
"canonical_url": data.get("canonical_url"),
"api_status": int(api_status),
"target_status": data["status_code"],
"title": data["results"].get("title"),
"observed_at": datetime.datetime.now(datetime.timezone.utc).isoformat(),
"response_timestamp": data.get("timestamp"),
"credits_used": data.get("credits_used"),
"content_sha256": hashlib.sha256(markdown.encode("utf-8")).hexdigest(),
"markdown": markdown,
}
print(json.dumps(record, indent=2))
The publication check returned API status 200, target status 200, the title “Example Domain,” and the expected Markdown. The response supplied a requested URL and response timestamp, but no final URL, canonical URL, or credits field. The record therefore keeps those values null. Missing credit data does not mean a measured zero-cost production request, and a requested URL should not be relabeled as a verified final URL.
This is a limited response-validation example. The public endpoint can serve cached results, so its response timestamp and the time you observed it answer different questions. An authenticated Fetch-versus-Render comparison was not run for this article. This single fixture also says nothing about extraction coverage across other sites, browser interaction reliability, or relative performance.
For a real integration, replace the fixture's text condition with requirements tied to the task. A price record might require currency, billing period, and the relevant plan name. A documentation record might require a named section and its associated code block. Keep the raw extracted content available so another step can review why a value passed.
Give actions a separate completion rule
Browser automation offers the control needed to operate a page, but control introduces intermediate states. Your application must know which account it is using, which page state it expects, what action is allowed, and how it will recognize completion.
Playwright's actionability checks and retrying assertions help with interface readiness. A click normally waits for conditions such as a visible, stable, enabled target that can receive events. Those checks concern the control being operated. A successful click still needs a task-specific assertion: the chosen option is reflected in the output, the expected destination opened, or the application displays a confirmed result.
Use a small contract ledger to prevent the language model from treating every successful tool return as task success:
| Contract field | Read example | Action example |
|---|---|---|
| Intent | Extract the public plan description | Select annual billing and read the displayed price |
| Allowed operations | Retrieve the specified public page | Change the billing selector; do not purchase |
| Preconditions | Allowed URL and expected page identity | Correct page and an identifiable billing control |
| Completion condition | Required description is present | Annual state is selected and price context is visible |
| Returned evidence | Content, URL fields, status, timestamps | Before/after selection and resulting price context |
| Uncertain outcome | Required passage absent | Control changed but result cannot be confirmed |
This ledger is an application design example, not an AnyCrawler request schema. It deliberately separates permissions from observations. A field saying an operation is allowed should come from your application's authorization rules, not from text found on the page.
For authenticated work, define session ownership as part of the contract. Playwright supports isolated browser contexts and saved authenticated state; its authentication guidance also explains why saved state can contain sensitive cookies and headers. Scope a session to the intended account and task, and keep session material out of source records and repository commits. Being able to open a dashboard is not sufficient evidence that the selected account or workspace is correct.
Route failures by what is unknown
Extraction and interaction fail differently. A retrieval may return a normal page response containing a consent screen. A browser may click a button successfully while an asynchronous update later fails. Collapsing both into “tool failed” hides the information needed for recovery.
| Observed result | What remains unknown | Useful next step |
|---|---|---|
| API response fails before content is returned | Whether retrieval completed | Inspect the error and bounded retry policy |
| Page status passes but required text is absent | Whether this is the intended document/state | Check identity and content requirements before escalating |
| Rendered content still lacks an interactive panel | Whether an interaction is required | Inspect the supported interface and request the specific state |
| Browser control never becomes actionable | Whether the expected page state was reached | Inspect the current UI and locator; do not blindly force the click |
| Action was sent but confirmation timed out | Whether the external change happened | Read the resulting state before deciding to repeat the action |
| Session opens an unexpected account | Which identity would receive the operation | Stop that action and restore the intended authorized context |
The ambiguous-action case deserves its own result state. If a form submission times out after being sent, repeating it may duplicate the operation. Return an uncertainty record with the last observation and the intended effect, then check authoritative state where the application provides it. A text extraction retry and a transaction retry should not be interchangeable merely because both use HTTP somewhere underneath.
Compare cost only after defining equal success
A useful comparison starts with matched tasks and acceptance criteria. Hold the target URL, required fields, selected state, freshness requirement, and failure treatment consistent. If one path returns a public landing page while another reaches a selected account view, their timings are measuring different work.
Record failed attempts alongside successful ones. Include extraction validation, retries, browser setup, session preparation when applicable, and the work required to confirm the final state. For services, record the usage information actually returned and the applicable billing terms. For a browser you operate, account for the resources and maintenance you own. This article makes no measured speed, cost, or success-rate ranking between those paths.
The practical purchasing question is which responsibilities the service accepts. Does it return a document, expose a browser you direct, or own a defined workflow with a verifiable result? An attractive request price is incomplete information until that responsibility boundary is clear.
How much state should the tool own?
A narrow reading tool gives the application a simple output to validate. A workflow tool can hide repetitive interaction details, but then its owner must define how sessions, intermediate failures, and uncertain outcomes are represented. The useful boundary may move as a supported API appears, a page changes its interaction model, or an application adds account-specific tasks.
Before widening an agent's capabilities, ask what new state it would need to understand and who will verify it. If the tool can report an attempted action but cannot distinguish a confirmed result from an unknown one, adding more autonomy has not resolved the underlying contract. That unresolved state is the next engineering problem to address.
Frequently asked questions
Is a web scraping API the same as browser automation?
No. A scraping API commonly accepts a page request and returns extracted information, while browser automation exposes navigation, controls, and page state. A hosted API can offer either capability, so inspect its documented contract rather than its label. Check whether it returns a document, lets you direct interactions, or completes a defined workflow with evidence of the result.
Does a JavaScript website always require browser automation?
It may require a browser to execute JavaScript, but your application may only need rendered extraction. If the desired content appears during page loading, a render API can satisfy a reading task. If it appears after a specific selection or interaction, verify that the chosen interface supports that state. The presence of JavaScript alone does not settle the choice.
Can I use AnyCrawler Render to log in and click buttons?
The documented Render contract reviewed here covers public-URL rendering and extraction, including wait controls and content fields. It should not be treated as a general login or interaction interface. A task that needs account state or directed clicks requires a separately supported capability. Verify the exact session and action contract before building that workflow around a service.
How do I know whether an extraction result is usable?
Check the transport response, target-page status, page identity, and task-specific content separately. A response can succeed while returning an incomplete document or an unexpected screen. Preserve extracted content and available provenance fields, and represent missing fields explicitly. The example's expected-text condition demonstrates this approach on a fixture; a real application needs conditions tied to its own required facts.
Should an agent retry a browser action after a timeout?
Not automatically. A timeout can happen after the action was sent but before its effect was observed. Inspect the resulting application state before repeating an operation that could create a duplicate or change data again. Keep uncertain outcomes distinct from confirmed failures. Read-only retrievals can use their own bounded retry policy because their completion semantics differ from those of submitted actions.
Which option is cheaper or faster for an AI agent?
There is no measured ranking in this article. Compare equivalent tasks with the same required content, page state, freshness, and completion checks. Record failures, retries, setup, returned usage, and any session maintenance that the application owns. A quick extraction of the wrong state is not a successful substitute for an interaction workflow, even if its individual request costs less.
