meraGEO's crawler

How it identifies itself

Every request meraGEO makes to a website sends this user agent:

Mozilla/5.0 (compatible; meraGEO/1.0; +https://merageo.com/crawler)

It does not run JavaScript, sign in, submit forms or read anything that is not public.

What it reads, and when

• When someone scans a domain: the homepage once, a request to check whether /llms.txt exists, and /robots.txt and the sitemap to find the site's pages. Nothing is fetched unless a person asked for that scan.
• For a domain on a meraGEO plan: the same again each week, plus the pages listed in its sitemap, up to the plan's page limit. At most four requests run at a time, and a page that has not changed is answered from its ETag or Last-Modified date without being downloaded again.
• When a subscriber asks for a rewrite: the page being rewritten.

robots.txt is read to find your sitemap. Because each visit is asked for by a person scanning that site, the crawler does not otherwise treat robots.txt as an instruction.

If your scan failed

A scan stops when your homepage answers with an error, such as 403 Forbidden or a challenge page, instead of the page itself. That is almost always bot protection doing its job. To let meraGEO read your site, allow requests whose user agent contains meraGEO:

• Cloudflare: under Security, then WAF, add a custom rule matching User Agent contains "meraGEO" with the action Skip, skipping the managed rules and Super Bot Fight Mode. The free plan's Bot Fight Mode cannot be skipped by a rule; switch it off while you scan, or ask us for help.
• Vercel: in the project's Firewall, add a custom rule that bypasses the user agent above, or turn off Attack Challenge Mode while you scan.
• Anything else (Akamai, Fastly, Sucuri, Wordfence, a hosting provider's firewall): allowlist the same user agent.

Requests come from cloud servers whose addresses change, so allow the user agent rather than an IP address. Then run the scan again. If your homepage is served from another address (for example www. or a country subdomain), scanning that address can also work.

Questions

Email hello@merageo.com with your domain if a scan still fails, or if you want us to stop reading your site. See also our privacy policy.