Crawler / Search / Page
Use page search when the target URL is unknown. It gives agents a shortlist of public pages before you spend crawl credits on the pages that matter. AnyCrawler runs the selected search channel, normalizes the response, and returns ranked result fields your agent can use before deeper crawling.
Search before crawl
Search keeps agents from crawling blindly. Start from a query, pick the right channel, and return normalized result collections that can become crawl targets, review queues, or research inputs.
curl
Run page search
curl -X POST "https://api.anycrawler.com/v1/search" \
-H "Authorization: Bearer $ANYCRAWLER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"channel": "page",
"query": "browser automation documentation",
"country": "us",
"page": 1,
"results_per_page": 10
}'Use the unified /v1/search endpoint and select the search type with the channel field.
Use cases
Treat page search results as a shortlist: dedupe domains, prefer authoritative sources, then call fetch for HTML-first pages or render for JavaScript-heavy results.
Start from a topic and collect candidate URLs before deciding which pages deserve fetch or render credits.
Find pricing, documentation, changelog, or competitor pages, then crawl the strongest results into Markdown.
When a known page moves or disappears, search for fresh alternatives and route the best result into the crawler.
How it works
The gateway handles request validation, account checks, search provider routing, response normalization, caching, and credit settlement.
Send a POST request with query, channel=page, and optional localization or pagination fields.
The gateway validates your API key, checks credits, applies cache policy, and sends the request to the selected search collection.
Upstream results are returned in a predictable response envelope with channel, query, nested search parameters, credits, and cache fields.
Use result links as crawl targets, visual review candidates, research inputs, or source lists for a larger agent workflow.
Response
Normalized page results
{
"ok": true,
"query": "browser automation documentation",
"credits_used": 20,
"cache_timestamp": 0,
"status_code": 200,
"results": {
"search_parameters": {
"channel": "page",
"country": "us",
"page": 1,
"query": "browser automation documentation",
"results_per_page": 10
},
"organic": [
{
"position": 1,
"title": "Playwright Documentation",
"link": "https://playwright.dev/docs/intro",
"snippet": "Fast and reliable browser automation documentation."
}
]
}
}Result collections vary by channel, but the public response envelope stays consistent.
What you get
The page search response is built for routing. Use snippets for triage, links for follow-up crawl requests, and credit/cache fields for production observability.
Request fields
The stable public contract focuses on the fields below. Undocumented passthrough fields are not part of new integrations.
| Field | Type | Required | Notes |
|---|---|---|---|
channel | "page", "images", "news", "videos", or "scholar" | Yes | Selects the search vertical on the unified public search endpoint. |
query | string | Yes | The search query to run. |
country | string | No | Optional country hint for localized search results. |
page | integer | No | Search result page to request. |
results_per_page | integer | No | Requested result count. Billing is calculated in blocks of 10 results. |
Response fields
AnyCrawler keeps the search envelope consistent while the result collection reflects the selected channel.
| Field | Meaning |
|---|---|
ok | Whether the gateway completed the search successfully. |
query | The query submitted in the request. |
credits_used | Credits settled for the request. |
cache_timestamp | Cache marker for reused responses. Search cache does not change pricing. |
status_code | The upstream search provider status code returned through the gateway. |
results.search_parameters.channel | The search vertical used for the request. |
results.search_parameters.country | The country hint used for localized search results. |
results.search_parameters.query | The query mirrored inside the normalized search parameters. |
results.search_parameters.page | The search result page returned. |
results.search_parameters.results_per_page | The requested result count used for billing. |
results.organic | Organic web results with titles, links, and snippets for source triage. |
JavaScript
Use the unified endpoint
const response = await fetch("https://api.anycrawler.com/v1/search", {
method: "POST",
headers: {
"authorization": `Bearer ${process.env.ANYCRAWLER_API_KEY}`,
"content-type": "application/json",
},
body: JSON.stringify({
channel: "page",
query: "browser automation documentation",
country: "us",
page: 1,
results_per_page: 10,
}),
});
const data = await response.json();
console.log(data.results);Change channel to switch between page, images, news, videos, and scholar search.
Search vs crawl
Treat page search results as a shortlist: dedupe domains, prefer authoritative sources, then call fetch for HTML-first pages or render for JavaScript-heavy results. Once you have promising result links, call fetch or render to turn selected pages into Markdown, metadata, links, and optionally visual evidence.
| API | Best for | Returns |
|---|---|---|
search | Finding candidate pages, sources, media, news, videos, or papers | Search result collections |
fetch | Fast, low-cost extraction when useful content is already in HTML | Markdown and page fields |
render | JavaScript-loaded pages that need browser execution | Rendered Markdown and page fields |
Errors
Use response status plus error_code and request headers to route retries, credit issues, authentication problems, and upstream provider failures.
Billing
Search requests settle by requested result blocks. Cache can improve response consistency, but search cache hits do not switch to crawl cache-hit pricing.
Pricing
Search requests use 20 credits per block of 10 requested results. The same AnyCrawler plans apply here.
Free
$0/mo
For early validation with one-time credits, 30 fetch requests per minute, and 1 browser concurrency slot.
Agent Lite
$5/mo
For low-frequency AI agents and occasional automations that need 15,000 monthly credits.
Builder
$20/mo
For builders shipping serious crawler workflows with higher limits and better unit economics.
Growth
$80/mo
For teams scaling production scraping with better unit economics.
Scale
$200/mo
For high-volume pipelines with pricing aligned to large ongoing usage.
FAQ
Short answers for teams using search to choose better crawl targets.
Use page search when the target URL is unknown. It gives agents a shortlist of public pages before you spend crawl credits on the pages that matter.
Yes. POST to /v1/search and change the channel field to page, images, news, videos, or scholar. Each channel also has this dedicated theme page for integration planning.
Search responses can be cached for speed and operational consistency, but search billing still follows the result-count formula rather than cache-hit pricing.
Use search to find candidate URLs, then call fetch or render on the strongest results when you need Markdown, metadata, links, or screenshots.