If WebStatBot shows up in your server logs, this page explains what it does and how to control it.
How to recognise it
WebStatBot sends this User-Agent:
WebStatBot/1.0 (+https://www.webstatbot.com/bot)
IP addresses. WebStatBot runs on several of our nodes and may use different IP addresses, which can change over time. Please recognise it by the User-Agent above, not by IP address, and do not rely on an IP allowlist.
robots.txt. WebStatBot respects robots.txt on every request. A path you disallow is never requested.
What it collects
WebStatBot collects only the technical attributes of a website that appear in the open data downloads on our homepage:
Domain — the host name and its country.
CMS — the content management system or site builder in use.
E-commerce — whether the site runs an online shop, on which platform, and where published: product count, EAN/barcode codes and prices. These are read from the same public product endpoints the e-commerce platform exposes itself (the standard storefront URLs each platform publishes for its own catalogue, e.g. a JSON/REST product listing) — never by logging in, scraping an admin area or bypassing any access control.
Marketing — the analytics, advertising and marketing tools the site loads.
Site language — the language the site declares or is written in.
Nothing outside this scope is collected for the published data. Results are published only as aggregate statistics.
How it crawls
Fetches data just in time: only at the moment a domain is analysed, only for the purpose of the analysis, and only from public URLs — either addresses found on the site itself or freely available public data about it. It does not build copies of websites.
The page's content is downloaded to read it once, to extract the attributes listed above (CMS, e-commerce platform, marketing tools, language, …) — the downloaded HTML itself is discarded straight after that and is never stored. Only those extracted attributes reach the database, and only as part of the aggregate statistics published openly on the homepage; there is no per-page or per-request archive to look up later.
Never logs in, submits forms, places orders, solves CAPTCHAs or bypasses any access control.
Follows robots.txt, including Crawl-delay.
At most one or two concurrent requests per site.
Backs off on 429 Too Many Requests and honours Retry-After.
Terms of collection
These terms describe the conduct of every request that identifies itself as WebStatBot.
Purpose. The data is collected solely to produce statistical and analytical information about the technology and language landscape of the web. It is not collected to profile, monitor or contact any person or business.
Public data only. WebStatBot reads only information that the site operator has made publicly accessible without authentication, payment or registration, and openly available data sets. Collecting, reading and analysing such data does not require the operator’s prior consent.
Technical signals are honoured. The crawler obeys robots.txt prohibitions and crawl-delay, the rate limits described above and HTTP status responses such as 403, 429 and 503. A machine-readable reservation of rights in robots.txt is treated as an opt-out.
No personal data. WebStatBot does not collect, store or publish personal data (PII). Where a fetched page incidentally contains such data, for example an e-mail address or a name, it is not extracted into the data set and is not retained. Only the attributes listed under “What it collects” are kept, and they describe websites, not people.
No onward disclosure. Collected data is not forwarded, sold, licensed or otherwise made available to any third party in raw form. The only output is the aggregate statistics published openly on this site under CC BY 4.0.
Legal basis. WebStatBot relies on the right to access and analyse lawfully accessible, publicly available content, including the text and data mining exception in Article 4 of Directive (EU) 2019/790, which applies where the rightholder has not reserved the use in a machine-readable way. It extracts facts and technical attributes, which are not protected by copyright, and in the sense of Directive 96/9/EC takes only insubstantial parts of databases, never the whole of or a substantial part of any database. Processing for statistical purposes follows Article 89 of Regulation (EU) 2016/679 and the principles of data minimisation and storage limitation.
Your rights are unaffected. You may opt out at any time using robots.txt (see below) or by writing to us. Nothing on this page limits any right granted to WebStatBot or to you by law, and nothing here is a waiver of any such right.