Suspicious Robot Sighting

Tue, Jun 3, 2008 at 9:16:42 am PDT

A web crawler using the IP address 66.165.190.122 has been hitting us like crazy for the past couple of hours, violating the rules in our robots.txt file and crawling through LGF as quickly as possible.

When I checked the access log, I discovered something that makes me a bit suspicious; all of the referring pages for this bot were from “right-wing” sites, and specifically from a limited group of anti-jihad sites including My Pet Jawa, Jihad Watch, and several others. Apparently someone is making an index of those sites for some unknown reason.

I blocked the IP in our htaccess file, and I suggest that if you run a blog you do the same.