How to Keep Duplicate Sources Out of a Research Agent
Group repeated text without losing original reporting, corrections, or provenance. Use conservative URL keys, validated snapshots, and a reviewable source ledger.
AnyCrawler guides
Practical guides to web scraping and AI agents.
Learn how to find web sources, extract readable content, and handle JavaScript pages in AI workflows. Browse guides by topic and choose the next task you need to solve.
Published guides
Production practices guides
14 guides
Retries, Timeouts, and Partial Failures in Web Access Pipelines
Keep useful documents when individual reads fail. Classify errors, bound retries and timeouts, validate content, and preserve an auditable partial result.
Canonical URL vs Final URL: Which One Should an Agent Cite?
Preserve requested, final, and canonical URL evidence, verify the cited passage before replacing its address, and handle redirects and missing API identity fields explicitly.
When Text Is Not Enough: Screenshot Evidence for AI Agents
A decision framework for AI agents that pairs machine-readable page content with screenshot evidence when layout, visible state, or human review changes the meaning of a claim.
Does Google News Have an API? Options for Monitoring Agents
Google retired its official News Search API. Compare RSS, third-party SERP services, dedicated news APIs, and an evidence-first monitoring workflow.
