Little Green Footballs - Buggy Diggbot Breaks the Rules

Show top rated links | LinkViewer

Buggy Diggbot Breaks the Rules

Mon, Apr 28, 2008 at 12:36:30 pm PDT

A robot from Digg.com has been rapidly running through everything at LGF, including images, with multiple hits per second. It’s doing this despite the following lines in our robots.txt file:

User-agent: *
Crawl-delay: 600

This rule is supposed to limit the amount of hits from all robots to no more than one every ten minutes, and the Digg crawler is blatantly ignoring it. From the files it’s requesting, it looks as if it’s out of control and misreading something.

Now blocked. The IP address of the ill-behaved bot: 64.191.203.34.

UPDATE at 4/28/08 1:47:15 pm:

Analyzed the logs some more and figured it out — there’s nothing nefarious going on. The crawler seems to have a bug; it is not correctly reading the BASE tag in our pages, and it was trying to find images in a nonexistent directory as a result. I fixed the problem by adding a couple of mod_rewrite rules to our htaccess file, and the Digg crawler is now unblocked.

(It was breaking the robots.txt rule, though, as it thrashed around trying to find files that didn’t exist.)

Hit the tip jar!

refresh | ► spinoff links (6)

Tags: Digg, Internet, Robots, Bots, Crawlers

86 comments

Comments are open and unmoderated, and do not necessarily reflect the views of Little Green Footballs.
Obscene, abusive, silly, or annoying remarks may be deleted, but the fact that particular comments remain on the site in no way constitutes an endorsement of their views by Little Green Footballs.
Posts that contain phone numbers, street addresses, email addresses or other personal information will also be deleted, as will posts that consist only of a variation on the word, "First!"
Comments that advocate violence will be cause for immediate banning with no appeal.
Disagreement and debate are welcome, but insults and abuse are not, and may cause your account to be blocked.
REMEMBER: posting comments at LGF is a privilege, not a right. Abuse that privilege, and your account will be blocked.

Hide comments | Jump to bottom

chinesearithmetic Mon, Apr 28, 2008 12:37:32pm

Digg Hits Bottom, Digs

Charlie Martel Mon, Apr 28, 2008 12:37:37pm

My God!

Charlie Martel Mon, Apr 28, 2008 12:38:02pm

What does all this mean? Seriously, I'm not a techie

Meremortal Mon, Apr 28, 2008 12:38:30pm

Bad bot, no doughnut.

Bubbaman Mon, Apr 28, 2008 12:38:54pm

The bigger question is why?

Charles Mon, Apr 28, 2008 12:39:26pm

re: #3 Charlie Martel

What does all this mean? Seriously, I'm not a techie

It looks like the crawler is trying to make a snapshot of the entire site.

zmdavid Mon, Apr 28, 2008 12:39:46pm

re: #3 Charlie Martel

What does all this mean? Seriously, I'm not a techie

Digg is downloading large chunks of LGF, sapping resources and slowing us down.

jcm Mon, Apr 28, 2008 12:40:13pm

Don't trust the digg robot.

marwan's daughter Mon, Apr 28, 2008 12:40:23pm

Ah, the intellectual fountain known as Digg.

Where stories against radical Islam, the excesses of the left, and even Scientology are buried by bury brigades.

I never liked it.

joncelli Mon, Apr 28, 2008 12:41:05pm

re: #6 Charles

Is this a weird sort of DOS attack or do they just want to grab the site and do something with it?

bosforus Mon, Apr 28, 2008 12:41:13pm

Big Digg is ripping LGF off!

galloping granny Mon, Apr 28, 2008 12:41:15pm

re: #6 Charles

It looks like the crawler is trying to make a snapshot of the entire site.

Strange. I wonder why. I hate robots that don't follow the rules.

Thanos Mon, Apr 28, 2008 12:41:27pm

Sure you don't mean 10 minutes? (600 seconds = 10 minutes)

winston06 Mon, Apr 28, 2008 12:41:44pm

re: #6 Charles

why would they wanna do it?

jcm Mon, Apr 28, 2008 12:41:48pm

re: #9 marwan's daughter

Ah, the intellectual fountain known as Digg.

Where stories against radical Islam, the excesses of the left, and even Scientology are buried by bury brigades.

I never liked it.

Well we do find out what is important to pale pasty residents of mama's basement.

ted Mon, Apr 28, 2008 12:42:00pm

BLAMMO !

winston06 Mon, Apr 28, 2008 12:42:10pm

re: #5 Bubbaman

Digg.com is infested with leftie kidz.

Lawrence Schmerel Mon, Apr 28, 2008 12:42:22pm

But, why?

bosforus Mon, Apr 28, 2008 12:42:36pm

Never trust a robot.

Charles Mon, Apr 28, 2008 12:42:41pm

re: #10 joncelli

Is this a weird sort of DOS attack or do they just want to grab the site and do something with it?

No, it's not an attack. I don't know what they're doing, but it's going through every file it can find and requesting nonexistent files as well. Acting a lot like a spam scraper, in fact.

Kragar (Proud to be Kafir) Mon, Apr 28, 2008 12:43:21pm

I never liked Digg

doppelganglander Mon, Apr 28, 2008 12:44:11pm

re: #14 winston06

why would they wanna do it?

Perhaps to cherry-pick at their leisure.

zmdavid Mon, Apr 28, 2008 12:45:25pm

Theft is the sincerest form of flattery.

ted Mon, Apr 28, 2008 12:45:56pm

Robots can be nasty.

Youtube Video

EC Marm Mon, Apr 28, 2008 12:46:25pm

What if that robot collected all of the intelligence of the great minds at lgf? It could take over the world.
"I don't think you want to post that, Dave"

Lawrence Schmerel Mon, Apr 28, 2008 12:46:26pm

I have a copy of the entire internet downloaded on a 3.5 floppy disc.

MI DB Mon, Apr 28, 2008 12:46:49pm

I Diggbot

OldLineTexan Mon, Apr 28, 2008 12:48:25pm

The Three Laws:

A robot may not injure a liberal cause or, through inaction, allow a liberal cause to come to harm.
A robot must not obey orders given to it by right-wing blogs, except where such orders would conflict with the First Law.
A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Charles Mon, Apr 28, 2008 12:48:28pm

Looked through the logs some more and I don't think there's anything nefarious going on.

I think it's out of control; it seems to be confused by the mod_rewrite trick I described a few days ago, and isn't properly using the BASE tag.

The crawler is using the PEAR HTTP_Request class, and it may have a bug.

shmuli[deleted] Mon, Apr 28, 2008 12:49:43pm

HelloDare Mon, Apr 28, 2008 12:50:00pm

Kick the bot in the nuts.

bosforus Mon, Apr 28, 2008 12:50:33pm

re: #29 Charles

Danger! Danger!

snowcrash Mon, Apr 28, 2008 12:50:39pm

It's not a bug, it's a feature.

Charles Mon, Apr 28, 2008 12:50:51pm

Sorry, shmuli - no phone numbers allowed. If you want to link to a whois lookup that's OK.

Lawrence Schmerel Mon, Apr 28, 2008 12:50:52pm

The internet is out of control! Run for your lives!

Cartman Mon, Apr 28, 2008 12:51:07pm

re: #26 Lawrence Schmerel

I have a copy of the entire internet downloaded on a 3.5 floppy disc.

You sure you got it all? It took me two floppies.

jcm Mon, Apr 28, 2008 12:51:11pm

Flying Pig Moment!Ex-Pink Floyd Bassist Roger Waters Unleashes School Bus-Sized Pro-Obama Inflated Pig at Coachella.

Kragar (Proud to be Kafir) Mon, Apr 28, 2008 12:51:29pm

re: #36 Cartman

You sure you got it all? It took me two floppies.

Got to zip it

shmuli Mon, Apr 28, 2008 12:51:54pm

Charles --

Sorry. Will do it right next time.

Did not mean to make more work for you.

Shmuli

Cartman Mon, Apr 28, 2008 12:52:58pm

re: #38 Kragar (Proud to be Kafir)

Got to zip it

Sheesh. I knew I forgot something.

zmdavid Mon, Apr 28, 2008 12:53:35pm

re: #27 MI DB

I Diggbot

DB - Diggbot?

Michigan Diggbot?

Lawrence Schmerel Mon, Apr 28, 2008 12:53:53pm

re: #36 Cartman

I haven't finished downloading littlegreenfootballs.com, yet.

The Other Les Mon, Apr 28, 2008 12:54:15pm

re: #31 HelloDare

Kick the bot in the nuts.

Take him to Detroit!

Youtube Video

CyanSnowHawk Mon, Apr 28, 2008 12:54:45pm

re: #42 Lawrence Schmerel

I haven't finished downloading littlegreenfootballs.com, yet.

Does someone keep blocking your robot?

Kragar (Proud to be Kafir) Mon, Apr 28, 2008 12:54:50pm

re: #37 jcm

OT

Flying Pig Moment!Ex-Pink Floyd Bassist Roger Waters Unleashes School Bus-Sized Pro-Obama Inflated Pig at Coachella.

Saw this report on it

the swine floated into the night sky. Waters said sadly and comically, "That's my pig."

Never to be seen again, hopefully BHO pulls the same act

yochanan Mon, Apr 28, 2008 12:55:06pm

re: #37 jcm

OT

Flying Pig Moment!Ex-Pink Floyd Bassist Roger Waters Unleashes School Bus-Sized Pro-Obama Inflated Pig at Coachella.

short bus?

Cartman Mon, Apr 28, 2008 12:55:09pm

re: #42 Lawrence Schmerel

I haven't finished downloading littlegreenfootballs.com, yet.

Bypass the Tag Storm, and it goes a lot quicker.

;)

Lawrence Schmerel Mon, Apr 28, 2008 12:55:39pm

re: #44 CyanSnowHawk

Do you know something that I don't?

Kulhwch Mon, Apr 28, 2008 12:55:54pm

re: #20 Charles

Acting a lot like a spam scraper, in fact.

That's what they get for buying their droids from Sand People.

}:) [ ... so to speak ... ]

taxfreekiller[deleted] Mon, Apr 28, 2008 12:55:54pm

Kragar (Proud to be Kafir) Mon, Apr 28, 2008 12:56:00pm

re: #47 Cartman

Bypass the Tag Storm, and it goes a lot quicker.

;)

Plus, turn off any porn filters when you get to the throbbing memo

taxfreekiller[deleted] Mon, Apr 28, 2008 12:56:39pm

Lawrence Schmerel Mon, Apr 28, 2008 12:58:05pm

re: #47 Cartman

You can get really stuck in that Tag Storm.

rabidsquirrel Mon, Apr 28, 2008 12:58:22pm

re: #28 OldLineTexan

The Three Laws:

A robot may not injure a liberal cause or, through inaction, allow a liberal cause to come to harm.
A robot must not obey orders given to it by right-wing blogs, except where such orders would conflict with the First Law.
A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

If only Asimov knew the horrible truth.

Cartman Mon, Apr 28, 2008 12:58:25pm

re: #51 Kragar (Proud to be Kafir)

Plus, turn off any porn filters when you get to the throbbing memo

True dat. The floppy drive doesn't like the throbbing memo.

RememberSekhmet? Mon, Apr 28, 2008 12:59:01pm

Hey guys, can you please knock it off?

rabidsquirrel Mon, Apr 28, 2008 1:00:10pm

re: #55 Cartman

True dat. The floppy drive doesn't like the throbbing memo.

I almost feel compelled to make a 'hard disk' joke...

Da_Beerfreak Mon, Apr 28, 2008 1:00:21pm

re: #29 Charles

Looked through the logs some more and I don't think there's anything nefarious going on.

I think it's out of control; it seems to be confused by the mod_rewrite trick I described a few days ago, and isn't properly using the BASE tag.

The crawler is using the PEAR HTTP_Request class, and it may have a bug.

So in other words, your custom code for the site has driven this poor Diggbot insane.
[insert evil laughter here]

CyanSnowHawk Mon, Apr 28, 2008 1:00:25pm

re: #48 Lawrence Schmerel

Do you know something that I don't?

In a thread about a runaway robot trying to download all of LGF, and Charles blocking the IP of said robot, you make a joke about not being finished downloading LGF, and you miss that reference?

I'm glad I'm not the only one that does things like that.

taxfreekiller[deleted] Mon, Apr 28, 2008 1:01:04pm

Cygnus Mon, Apr 28, 2008 1:02:52pm

re: #19 bosforus

Never trust a robot.

Now, that is one weird sci-fi series. Never heard of it before.

Cartman Mon, Apr 28, 2008 1:03:27pm

re: #58 Da_Beerfreak

So in other words, your custom code for the site has driven this poor Diggbot insane.
[insert evil laughter here]

Considering the bot's source, it was probably 3/4 of the way there already.

Ben Hur Mon, Apr 28, 2008 1:05:16pm

IDF: Palestinian family killed by terrorist bomb

Wasn't Israel.

But the damage is already done.

The Other Les Mon, Apr 28, 2008 1:05:36pm

re: #60 taxfreekiller

Understand, lgf's is hurting a loon commie, one the msm and other power mad commies like Soros wants in the White House, its not
a game to these power hungry loons.

I can't repeat and emphasize enough the fact that for this bunch:

POWER IS LIFE!

Lawrence Schmerel Mon, Apr 28, 2008 1:05:39pm

re: #59 CyanSnowHawk

I have no idea what you are talking about. I am going back to work. I'm going to copy this sucker if I have to do it one page at a time.

hayseed Mon, Apr 28, 2008 1:06:10pm

re: #61 Cygnus

Now, that is one weird sci-fi series. Never heard of it before.

I have an Angela Android

adult ears

taxfreekiller[deleted] Mon, Apr 28, 2008 1:06:27pm

Silhouette Mon, Apr 28, 2008 1:06:37pm

re: #49 Kulhwch

That's what they get for buying their droids from Sand People.

Jawas, jawas, jawas, jawas!

Uh, did I just expose my inner-geek?

jcm Mon, Apr 28, 2008 1:06:38pm

re: #63 Ben Hur

IDF: Palestinian family killed by terrorist bomb

Wasn't Israel.

But the damage is already done.

I told junior not to bring work home.

Cartman Mon, Apr 28, 2008 1:06:55pm

Well, time to take a spin on the new Harley. Better get it in while I can. It's supposed to snow here tomorrow. Global warming my ass. Hrrumph.

Fat Jolly Penguin Mon, Apr 28, 2008 1:08:18pm

re: #68 Silhouette

Jawas, jawas, jawas, jawas!

Uh, did I just expose my inner-geek?

Nah, you'd be doing that if you asked if the Diggbot speaks Bocce.

Ben Hur Mon, Apr 28, 2008 1:11:16pm

re: #63 Ben Hur

IDF: Palestinian family killed by terrorist bomb

Wasn't Israel.

But the damage is already done.

It's amazing how fast YNET goes back and edits and changes Ali Waked's articles once the truth comes out.

This has happened more than just a few times.

He reports what his "souces" tell him, then it comes out that it was complete BS, and they quickly and quietly change the article online.

Is that normal?

LeftJustAintRight Mon, Apr 28, 2008 1:14:21pm

Union Bots Never do more work than coded
Same with Union Workers
Charles
Unionize the bots and every 10 minutes they will need a 20 minute break
LOL

Cap'n DOC Mon, Apr 28, 2008 1:18:02pm

re: #26 Lawrence Schmerel

Are you an Al Gore Foolower?

Lawrence Schmerel Mon, Apr 28, 2008 1:21:17pm

re: #74 Cap'n DOC

You know, you have Al Gore to thank for the fact that you can use your computer to ask me that question.

Charles Mon, Apr 28, 2008 1:48:48pm

See update above - nothing bad about the Diggbot, it has a bug. And it was ignoring robots.txt.

Lawrence Schmerel Mon, Apr 28, 2008 2:14:27pm

re: #76 Charles

You sure know how to ruin a good comment thread.

madjadbad Mon, Apr 28, 2008 2:26:30pm

Interesting. I think Digg maintains a mirror site of the submitted links because if the article becomes popular, the traffic deluge often causes the original link's server to have a heart attack.
It sounds like the mirroring code may be buggy.
I did a search at Digg for "littlegreenfootballs" and it looks like almost every story on LGF has been dugg at least once, but the ones with the most diggs in the last 2 days only have 32 diggs. None of them made the front page.
I don't know how many diggs it takes before they mirror the story.

NomadOfNorad Mon, Apr 28, 2008 2:30:36pm

On a different note... I heard a coupla months back about a new rival to Digg, that did the same sorts of vote-for-a-site stuff that Digg does, but didn't have some of the flaws of Digg... but I couldn't remember the name of the place. So, I did a Google search for "digg rival" and it turns out there's more than one wanna-be Digg out there.

I think Mixx might have been the one I'd heard about, though, but there's also one called NewsPond, and another called Reddit, and it seems Netscape (now calling themselves Propeller) have come back as a Digg competeter, too. Even Yahoo! seem to be getting into the act with their My Web 2.0. And there's another thing called Drupal or Drigg or something of the sort, apparently.

Wow!

I suppose LGF should add at least one of these new guys to the set of buttons we can click on to call attention to specific articles like we do now with Digg and de.lic.ious and stuff. Problem is, where do you stop in adding these guys? (There are probably even more of them out there that I haven't found in that cursory search.)

BingoBunny Mon, Apr 28, 2008 2:39:38pm

Damn robots.. next they'll want the vote.

Spiny Norman Mon, Apr 28, 2008 2:40:23pm