TrafficWonkBot

TrafficWonkBot crawls domains to collect the following signals:

• Page titles, meta descriptions, H1 headings, and structured data (schema.org)
• Internal link graph — all links between pages on the same domain
• External link signals — outbound links and their anchor text
• HTTP response codes, redirect chains, canonical tags
• robots.txt and sitemap.xml presence and compliance
• Page speed proxy (response time) and HTTPS status
• Indexability signals (noindex, nofollow meta tags)

TrafficWonk uses the bot in bounded, purpose-specific jobs. A domain may be fetched because someone requested a free visibility check, because its owner added or claimed it, or because it appears in a public listing used for discovery. A crawl does not imply that the site's owner has claimed or verified its TrafficWonk profile.

• It does not log in to private, password-protected or authenticated areas.
• It does not submit forms, make purchases or take actions on behalf of visitors.
• It does not solve CAPTCHAs or defeat access controls.
• It does not use residential proxies, rotate IPs or disguise its declared identity.

The figures below describe a full-site link-analysis crawl. Single-page checks and public-listing discovery use smaller, bounded jobs.

• robots.txt is fetched and evaluated for every crawl job and cached for up to 6 hours. All Disallow and Allow directives are respected. Crawl-delay is honoured.
• For full-site link crawls, a robots.txt rule that disallows TrafficWonkBot on the root path aborts the crawl before it follows links.
• HTTP 429 Too Many Requests is respected: the Retry-After header is honoured before retrying.
• A WAF block is recorded. Full-site crawls back off for 30 seconds after the first block, 120 seconds after the second and stop after three consecutive blocks.
• No residential proxies, no IP rotation, no burst requests.

TrafficWonkBot runs on Railway. Railway does not publish a guaranteed outbound IP range for all services, so outbound addresses may vary by deployment region and plan.

We do not currently publish a fixed IP allowlist. The stable identifier is the declared user-agent shown above. That user-agent is visible and can be spoofed, so it is not an authentication mechanism. If you choose to allow the bot through your WAF, limit the rule to public content and use the narrowest exception you are comfortable with.

We do not use residential proxies, IP rotation, or any technique designed to obscure the origin of our requests.

To allow full access, no action is required — TrafficWonkBot is permitted by default unless you have a specific Disallow rule. To explicitly allowlist the bot:

User-agent: TrafficWonkBot
Allow: /

To block TrafficWonkBot entirely:

User-agent: TrafficWonkBot
Disallow: /

If you control the WAF and want to permit TrafficWonkBot, create a narrow exception for public content only. TrafficWonk does not ask site owners to disable bot protection globally. Because the user-agent is visible and can be spoofed, treat this as an owner-controlled trust decision rather than proof of identity.

(http.user_agent contains "TrafficWonkBot")

Apply the narrowest action and path scope you are comfortable with. Never use a bot exception to expose private or authenticated content.

User-Agent:      TrafficWonkBot/1.0 (+https://trafficwonk.com/bot; contact: bot@trafficwonk.com)
Accept:          text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-GB,en;q=0.9
Accept-Encoding: gzip, deflate, br
Connection:      keep-alive
From:            bot@trafficwonk.com

These are the headers used by programmatic HTTP requests. Where a site needs JavaScript to render ordinary public content, TrafficWonk may use a headless browser renderer. It does not solve CAPTCHAs or access authenticated pages.

• robots.txt — add User-agent: TrafficWonkBot and Disallow: / (see above). The bot will stop at the next robots check.
• Direct request — email bot@trafficwonk.com with your domain. We will remove it from the crawl queue within 24 hours.