Crawler information

AnswerMentionsBot

AnswerMentionsBot fetches a sample of a website when someone requests a technical SEO audit of it on AnswerMentions, and for a full report it reads the few pages that rank above that site on Google. It never crawls on its own schedule.

User agents

Both contain AnswerMentionsBot, so one rule matches both. It never sends a plain browser or Googlebot user agent.

Main user agent

Mozilla/5.0 (compatible; AnswerMentionsBot/1.0; +https://answermentions.com/bot)

Mobile — one extra request for the audited page, to compare mobile and desktop HTML

Mozilla/5.0 (Linux; Android 14; Mobile) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Mobile Safari/537.36 (compatible; AnswerMentionsBot/1.0; +https://answermentions.com/bot)

What it requests

/robots.txt
Up to 6 sitemap files (those in robots.txt, /sitemap.xml and /sitemap_index.xml) plus up to 5 child sitemaps of one index
Up to 20 HTML pages for a free report, 40 for a full one — found through your links and a sample of your sitemap
Full reports only: up to 8 of your pages that rank on Google but weren’t reached by the crawl, one request each, plus robots.txt once more for each host they’re on
Variants of the audited URL — http / https, www / apex, trailing slash, upper case, /index.html and a utm-tagged copy — to check redirects
One random path such as /answermentions-404-probe-1a2b3c4d, to check that missing pages return 404
The start of up to 20 sitemap URLs it didn’t crawl and up to 11 canonical targets — enough to read status, noindex and canonical
HEAD requests to up to 30 internal link targets it didn’t crawl, to find broken links
HEAD requests to up to 20 images and the og:image, to read their file size
HEAD requests to up to 30 links pointing to other sites (5-second timeout), to find dead outbound links

Pages are read as raw HTML, up to 2 MB each. It never runs JavaScript, submits forms or logs in, and doesn’t download scripts, stylesheets or images. When a HEAD request fails it may retry with a GET that stops after the first 4 KB.

In all, at most 189 requests to your site for a free audit and 245 for a full one, spread over a minute or two — that ceiling assumes every HEAD request is retried as a GET and every page-request slot is used; most audits make far fewer. Redirects add one request per hop (up to 5). Separately, the outbound-link check sends at most 60 requests to the other sites your pages link to, and reading the certificate takes up to 3 TLS handshakes (no HTTP request).

How politely

Crawls pages 4 at a time and checks links 4 at a time; page fetches and link checks on the same host start at least 250 ms apart
Follows the robots.txt rules for AnswerMentionsBot (or *) for every page it crawls, every sitemap URL or internal link it samples, and every top-ranking page it reads for a full report
If robots.txt can’t be read — a 5xx error, a 429 or no response — it treats the whole site as disallowed (RFC 9309): no page beyond the URL the audit was requested for, exactly as with Disallow: / below
Honours Crawl-delay while crawling pages, up to 2 s; a longer delay is capped at 2 s and the number of pages is halved
Stops on a 429, a challenge page or 403 responses and skips the remaining checks on your host. If the audited URL itself is refused, the audit stops there and the report says the site blocked us. It never retries with a different user agent.
Free audits: at most 6 fresh runs of the same site per hour across all users, and the same URL audited again within 6 hours reuses the earlier result unless a re-run is forced. Paid reports can also be re-run for 30 days after purchase.

Top-ranking pages (full reports)

A full report compares the audited site with the pages that outrank it. For up to 6 of that site’s search queries we take the top 5 organic Google results (via DataForSEO), and AnswerMentionsBot fetches each of those pages once per report run:

It requests that host’s /robots.txt first and skips the page if AnswerMentionsBot (or *) is disallowed from it — or if robots.txt can’t be read
Then one GET for the page itself — raw HTML up to 2 MB, 8 s timeout. It doesn’t follow links on the page or fetch its images, scripts or stylesheets
At most 30 pages per report run, across all sites, 4 at a time; requests to the same host start at least 250 ms apart, longer if robots.txt sets a Crawl-delay

So if one of your pages ranks for a query an audited site cares about, you may see one request for it plus your robots.txt — nothing else on your site. A re-run of that paid report fetches the same pages again without a new search, unless the site’s owner has added a new target keyword. Blocking AnswerMentionsBot in robots.txt stops it.

Two parts of a full report send no request to any website. The brand-reputation check is two Google searches run through DataForSEO — the audited site’s brand name, and the name plus “reviews”. Search Console data, read only when the audited site’s owner connects it, comes from Google’s API. Neither shows up as AnswerMentionsBot in anyone’s logs.

How to block it

Stop the crawl with robots.txt:

User-agent: AnswerMentionsBot
Disallow: /

AnswerMentionsBot then fetches none of your pages beyond the URL the audit was requested for. Each audit still makes the site-level requests above: robots.txt, sitemaps, that URL with its variants and canonical target, the 404 probe, and size checks on that page’s images.

Refuse every request by blocking user agents containing AnswerMentionsBot at your firewall or CDN.

The audit then reports the site as blocked instead of giving it a score. Neither option affects Google.

Contact

Questions, or traffic you didn’t expect? Email [email protected] with your domain and the time of the requests, and we’ll look into it.

Requests come from Vercel’s cloud and the IP range isn’t fixed, so please match on the user agent. The PageSpeed Insights part of an audit is run by Google: those visits come from Google’s servers with a Chrome-Lighthouse user agent, not from AnswerMentionsBot.