Tech Note: Unusual Robot Surge Detected
Charles Johnson
Tue May 20, 2008 at 3:25 pm PDT • Views: 316
All day I’ve been watching the online visitor count suddenly jump way up, but without the referrals that would indicate a link from a high volume site. It’s happened four or five times, and the cause in each case has been a web crawler trying to load every page at the site as quickly as possible, with dozens of hits per second. I’ve been blocking them as they happen, and so far the IP addresses have traced to:
United Arab Emirates
China (a whole block of addresses)
Vietnam
Netherlands
The China bot was using powerlineblog.com as a referrer, probably as a weak disguise.
Something’s going on out there on the innernut.
Frank says: