Web Index

The TrafficWonk Web Index

TrafficWonk maintains a large corpus of known sites and domains, and independently observes a growing subset of them to establish current evidence of what is actually serving. Knowing that a site exists and having checked it are two different things, and TrafficWonk keeps them apart.

Current scale

Known Sites
Observed by TrafficWonk
Serving
Substantive

Known sites, exactly: —

Live figures are being refreshed. The Web Index contains more than 262 million known sites; TrafficWonk independently observes a growing subset of them.

Web Index snapshot — · —
Observations — · last run —

Known is not the same as observed

Every figure above means something specific, and the difference between them is the point of the architecture rather than a caveat hidden in a footnote.

Known
—
The site or domain exists in trusted web evidence. It has been seen, linked to or recorded — not necessarily visited by us, and not necessarily serving today.
Observed by TrafficWonk
—
TrafficWonk has independently checked it: resolved its DNS, attempted a connection and recorded what actually happened, at a known point in time.
Serving
—
TrafficWonk obtained current evidence that a web service responded. Responding is not the same as having a website worth reading.
Substantive
—
TrafficWonk observed evidence of meaningful website content, rather than a parked page, a placeholder, an error page or a holding redirect.
Alongside the Web Index
Published Profile
— in the Directory
TrafficWonk has selected the site and enriched it far enough to publish a public visibility profile. Profiles come from their own enrichment pipeline, so they are not a subset of the observed sites above.

Why the distinction matters

Large web datasets are not lists of working websites. A single corpus will happily contain domains that were merely linked to from somewhere, domains observed years ago and never since, sites that genuinely serve a business today, parked pages waiting to be sold, redirects to somewhere else entirely, and hosts that cannot be reached at all.

Collapsing all of those into one “websites” number produces a big figure and a misleading one. TrafficWonk deliberately does not do that. The Web Index records what is known; observation records what was actually found, when, and on what evidence.

The honest version of the headline. The Web Index contains more than 262 million known sites. TrafficWonk has independently observed — of them so far. Those are different claims, and we keep them that way.

How TrafficWonk learns about the web

01 · Web Index
Trusted web evidence
A large-scale corpus of known sites and domains, assembled from trusted public web evidence including Common Crawl. This establishes what exists.
02 · Observations
TrafficWonk checks it
TrafficWonk independently gathers current factual evidence about selected sites, using its own declared crawler. This establishes what is true now.
03 · Intelligence
Evidence becomes insight
That evidence feeds visibility profiles, search intelligence, AI visibility intelligence and the other capabilities TrafficWonk offers.

Common Crawl is evidence that feeds the Web Index. It is not the source of TrafficWonk's observations: those are gathered independently by TrafficWonkBot, TrafficWonk's declared crawler, which respects robots.txt and publishes its behaviour.

Evidence, not guesswork

When TrafficWonk observes a site it does not immediately reduce the result to “alive” or “dead”. A domain can resolve but refuse connections. It can serve a page that is really a holding notice. It can be perfectly healthy and simply decline to talk to automated clients. Those are different facts and they are kept as different facts.

Each observation preserves dimensions such as:

DNS evidence Web response & transport Redirect behaviour Content state Blocking Observation time

Keeping the dimensions separate is what lets TrafficWonk say something precise later: that a site was reachable but restricted, or that it served a parked page, rather than flattening both into a single misleading verdict.

Related

TrafficWonkBot — crawler policy and allowlisting
The TrafficWonk Visibility Directory
Company information