AI scrapers hit one site 214 times for every human visit
The AI economy's quiet cost showed up in one publisher's server logs this week: a 1.5-million-page site found bots outnumbering humans by more than 200 to one — and its analytics dashboard never saw them.
A year of server logs from PatronView, a 1.5-million-page philanthropy research site, shows AI crawlers now generate more than 99% of its traffic — roughly 214 non-human page loads for every human one. Founder Nick Gray published the raw numbers on August 7 after a year of fighting scrapers, and they're the most concrete account yet of what the crawl-to-referral gap actually looks like on a real site. In the week he measured, the server answered 2.5 million requests and served 1.28 million full pages, while his self-hosted Plausible analytics recorded just 5,977 human pageviews. The breakdown is stark: Anthropic's Claude-SearchBot crawled about 35,000 pages per referral sent back, and Amazon's Amzn-SearchBot hammered the site with up to 117,000 requests a day while referring zero visitors — Gray eventually blocked it outright with Cloudflare firewall rules.
The gap between what the logs show and what dashboards report is structural, and that's the part every publisher should care about. JavaScript-based analytics tools like Plausible, Fathom, and Google Analytics only count visitors whose browsers execute JavaScript — and almost no bot does, so a site owner checking the usual dashboard has no idea what's actually hitting the server. Cloudflare's own Bot Management dashboard, opened July 1, documents crawl-to-referral ratios spanning from 118 to nearly 50,000 across the crawlers it tracks, and its Radar data shows bots passed humans in 2026 (57.5% of traffic to 42.5%), with most AI crawling aimed at training, not search — meaning it sends almost no readers back.
Here's the nuance worth holding onto: Gray's numbers actually corroborate Anthropic's claim that its crawlers respect robots.txt. Claude-SearchBot's request volume dropped after his firewall change, and Anthropic's updated crawler documentation (February 2026) spells out the separate roles of ClaudeBot, Claude-User, and Claude-SearchBot. The deeper problem isn't defiance — it's the broken value exchange of the web itself. Traditional search crawled a page and sent readers back; AI crawlers take the content and return almost nothing, and the sites producing that content are increasingly blocking the door. Expect more publishers to follow PatronView's playbook, and more licensing fights as the crawl-to-refer math keeps getting worse.
What to watch: whether the big labs start paying for crawl rights the way they've started paying for news licensing — and whether robots.txt, a 30-year-old gentlemen's agreement, survives contact with the training-data gold rush.
If a crawler takes your content and sends back zero readers, is that fair use or the web's biggest unpaid bill? Tell us in the comments.
Sources: Techmeme · PPC Land · Cloudflare · Kinsta
