TagsTagsA robot from Digg.com has been rapidly running through everything at LGF, including images, with multiple hits per second. It’s doing this despite the following lines in our robots.txt file: User-agent: * Crawl-delay: 600 This rule is supposed to limit the amount of hits from all robots to no more than one every ...