Robots Crawling All Over Us

Wed, Aug 13, 2008 at 3:53:12 pm PDT

In the past few hours, I’ve discovered and blocked at least six web crawlers trying to go through the site and download everything as quickly as possible, in violation of the rules in our robots.txt file. Two of them seemed to be focusing on the posts related to the Georgia crisis.

Pretty unusual activity for a Wednesday afternoon.