Crawler

What is this resource?

polite-crawler: a polite web crawler for research and AI agents. Fetch once, keep it, read it many times. robots.txt, one request per site, full-text search, MCP server.

Review the original project for installation steps, licenses, compatibility and updates. This directory provides discovery links, not downloadable or verified code.

Safety & attribution

This is an external resource, not independently security-audited by this directory. Inspect permissions, scripts and changes before using it with your files or accounts.