The MangoSearch crawler

If you saw MangoSearch in your logs, this page is for you. It is a small, polite crawler that feeds a curated research index — it does not train AI models and it does not resell your content.

How to recognise it

User-Agent: MangoSearch/0.1 (curated research index; +https://mymango.app/)

The robots.txt product token is MangoSearch.

What it fetches

• Only sites that are on a curated list — never a blind crawl of the web.
• Mostly feeds and pages linked from them, plus a shallow sample of each site.
• Just the HTML (and feeds). No images, scripts, form submissions or logins.

How it behaves

• robots.txt is obeyed. Rules for MangoSearch take priority over *. Crawl-delay is honoured.
• At most one request in flight per host, and at least 2 seconds between requests to the same host (longer if your Crawl-delay says so).
• It backs off on 429/503 and stops visiting a host that keeps refusing it.
• Robots meta tags and X-Robots-Tag headers (including ones scoped to mangosearch) are read on every page; nofollow is honoured.
• A strict global bandwidth cap and a hard limit on the number of pages fetched.

Opting out

To keep MangoSearch off your whole site, add this to /robots.txt:

User-agent: MangoSearch
Disallow: /

To keep one section out, disallow just that path. robots.txt is cached for at most 24 hours, so a new rule takes effect within a day.