About the crawler

How to recognise it

WebStatBot sends this User-Agent:

WebStatBot/1.0 (+https://www.webstatbot.com/bot)

What it collects

• Public web statistics: the technologies and platforms a site uses, page structure, availability.
• E-commerce data: whether a site runs a shop, its platform, public product listings and prices.

It reads only publicly accessible pages. It does not log in, submit forms or place orders. Aggregated results are published as open data.

How it crawls

• Follows robots.txt, including Crawl-delay.
• At most one or two concurrent requests per site.
• Backs off on 429 Too Many Requests and honours Retry-After.

How to block it

To stop WebStatBot crawling your whole site:

User-agent: WebStatBot
Disallow: /

Or only part of it:

User-agent: WebStatBot
Disallow: /private/
Crawl-delay: 10

Changes are picked up on the next visit.

Contact

Problems, questions or a request to remove data: bot@webstatbot.com.