LGF Front Page > TagViewer (entries by tag)

1 entry, in 1 page    per page

Tag(s):

Separate tags with commas. Tags may contain more than one word.

Date range:

Buggy Diggbot Breaks the Rules

Mon, Apr 28, 2008 at 12:36 pm PDT

A robot from Digg.com has been rapidly running through everything at LGF, including images, with multiple hits per second. It’s doing this despite the following lines in our robots.txt file: User-agent: * Crawl-delay: 600 This rule is supposed to limit the amount of hits from all robots to no more ...

LGF, Digg, Internet, Robots, Bots, crawlers