How to Extract Web Tables Without Losing Headers, Units, and Sources
Preserve table header paths, raw cell text, units, notes and sources before normalizing web data. Includes tested acquisition and parsing examples plus review rules.
AnyCrawler guides
Practical guides to web scraping and AI agents.
Learn how to find web sources, extract readable content, and handle JavaScript pages in AI workflows. Browse guides by topic and choose the next task you need to solve.
Published guides
Web search guides
15 guides
Web Crawler API vs Page Fetch API: Do You Need Discovery or One URL?
Choose a single-page Fetch, query-driven Search, or bounded site crawl by the URLs you know and the coverage you need. Includes a tested public probe and verification checklist.
How to Build a Competitive Intelligence Agent from Public Web Sources
Define comparable entities and fields, preserve public source evidence, and separate observed changes from business interpretations. Includes a tested collector and review rules.
How to Find and Verify Research Papers with Scholar Search
Match paper identity, distinguish preprints from published versions, and preserve the passage behind each citation. Includes a BERT example and tested page collector.
How to Build a News Monitoring Agent with Search, Render, and Screenshots
Build a reviewable news monitoring workflow with source states, separate timestamps, correction handling, and optional screenshot evidence. Includes a tested collection probe.
Video Search for Research Agents: When the Surrounding Page Matters
Build a video research workflow that separates search candidates, source-page context, transcripts, and reviewed footage before an agent cites a claim.
How Research Agents Should Verify Image Sources
Match an image candidate to its source page, caption, and credit. Build a claim-scoped evidence record that keeps missing context and provenance questions visible.
How to Keep Duplicate Sources Out of a Research Agent
Group repeated text without losing original reporting, corrections, or provenance. Use conservative URL keys, validated snapshots, and a reviewable source ledger.
Search API vs Scraping API: Discovery and Extraction Are Different Jobs
Choose Search for source discovery and a scraping or page-read API for known-URL extraction, then connect them with a verifiable handoff contract.
