Tech Note: Unusual Robot Surge Detected

Tue, May 20, 2008 at 3:25:19 pm PDT

All day I’ve been watching the online visitor count suddenly jump way up, but without the referrals that would indicate a link from a high volume site. It’s happened four or five times, and the cause in each case has been a web crawler trying to load every page at the site as quickly as possible, with dozens of hits per second. I’ve been blocking them as they happen, and so far the IP addresses have traced to:

United Arab Emirates
China (a whole block of addresses)
Vietnam
Netherlands

The China bot was using powerlineblog.com as a referrer, probably as a weak disguise.

Something’s going on out there on the innernut.