Crawler / Search
Choose a search channel before crawling.
Search is the discovery layer for AnyCrawler. Start from a query, pick the channel that matches the source type, then route the best results into fetch, render, or screenshot.
Search channels
Pick the source type first
Each channel keeps the same request envelope but returns result collections tuned for a different discovery job.
| Channel | Best for | Follow-up |
|---|---|---|
page | General web source discovery and crawl seed lists. | Fetch or render selected result links. |
images | Visual candidates, source pages, and media research. | Crawl source pages or capture screenshots for provenance. |
news | Fresh coverage, launches, incidents, and monitoring workflows. | Render or screenshot sensitive stories for verification. |
videos | Demos, talks, tutorials, interviews, and video-first results. | Crawl surrounding page context when text is needed. |
scholar | Research papers, citations, and technical references. | Fetch abstracts, papers, references, or author pages. |
Search to crawl
Use results as a shortlist, not the final extraction
Search tells the agent where to look. When a result deserves deeper handling, use fetch for HTML-first pages, render for JavaScript pages, or screenshot when visual evidence is part of the record.
Page Search API Find public web pages and source URLs before crawling. Images Search API Find visual candidates, image URLs, and source pages. News Search API Track fresh coverage and recent sources for monitoring workflows. Videos Search API Find video-first sources, demos, tutorials, and review candidates. Scholar Search API Find papers, citations, and research references for agents.