How to Extract JSON-LD from Web Pages and Verify Its Claims
Inventory embedded JSON-LD with a tested Python example, select the right entity, and check extracted fields against readable page evidence.
AnyCrawler guides
Practical guides to web scraping and AI agents.
Learn how to find web sources, extract readable content, and handle JavaScript pages in AI workflows. Browse guides by topic and choose the next task you need to solve.
Published guides
Latest guides
33 guides
How to Check robots.txt Before an AI Agent Fetches a Web Page
Check a page's robots rules for the actual crawler identity before an agent reads it. Use an explicit allow, deny, or defer decision and preserve the evidence across redirects and failed rule requests.
How to Chunk Web Pages for RAG Without Losing Citations
A practical way to split retrieved web pages into RAG chunks while keeping section context, source identity, and citations verifiable.
How to Extract Web Tables Without Losing Headers, Units, and Sources
Preserve table header paths, raw cell text, units, notes and sources before normalizing web data. Includes tested acquisition and parsing examples plus review rules.
Web Crawler API vs Page Fetch API: Do You Need Discovery or One URL?
Choose a single-page Fetch, query-driven Search, or bounded site crawl by the URLs you know and the coverage you need. Includes a tested public probe and verification checklist.
How to Design an Agent-Safe Web Access Tool Contract
Design narrow agent web tools with validated inputs, bounded content, URL identity, explicit errors, and usage fields. Includes a tested public-page adapter.
How to Evaluate Web Extraction Quality Before Sending Content to an LLM
Check extracted documents for missing facts, sidebar noise, broken structure, and unknown provenance before admitting them to model context.
Screenshot Evidence for Compliance and Human Review Workflows
Build a reviewable screenshot evidence bundle with source URLs, separate capture times, companion text, file hashes, and explicit human review decisions.
Monitoring Pricing Pages Without Losing the Original Evidence
Keep pricing observations comparable with billing and currency context, original text, screenshots, and review states. Includes a tested collector and evidence record.
