Robots Crawling All Over Us

Charles Johnsonfollow me on twitter
Wed Aug 13, 2008 at 3:53 pm PDT • Views: 225

In the past few hours, I’ve discovered and blocked at least six web crawlers trying to go through the site and download everything as quickly as possible, in violation of the rules in our robots.txt file. Two of them seemed to be focusing on the posts related to the Georgia crisis.

Pretty unusual activity for a Wednesday afternoon.

Advertisement

220 comments

^ back to top ^

Name:

Pass:

Register Forgot Your Password? Re-send Confirmation (To log in, cookies must be enabled in your browser!)

Turn off ads by subscribing!
For about 33 cents a day, our subscription option turns off all advertisements at LGF!
Read more...


► LGF Headlines

  • Loading...

► Tweeted Articles

  • Loading...

► Tweeted Pages

  • Loading...

► Top 10 Comments

  • Loading...

► Bottom Comments

  • Loading...

► Recent Comments

  • Loading...

► Tools/Info

► Tag Cloud

► Contact

You must have Javascript enabled to use the contact form.
Your email:

Subject:

Message:


Messages may be published in our weblog, unless you request otherwise.
Tech Note:
Using the Contact Form

More Partners

Compare Electricity Prices in your area. Texas Electricity is deregulated; you have the right to choose Texas Electric Rates from among many Texas Electric Companies.

Why can't I just eat my waffle?

TwitterFacebook
LGF Pages
Recent Pages

Channeling Confucius
Astronomy Picture of the Day: Looking Back at an Eclipsed Earth
38 minutes ago
Views: 28 • Comments: 0
Tweets: 0 • Rating: 0

Aigle
Ha'aretz, Lost in Translation
2 hours, 54 minutes ago
Views: 90 • Comments: 3
Tweets: 0 • Rating: 0

Randall Gross
Government Employment Drops Under Obama, but Media Run With Romney Myth Anyway
4 hours, 44 minutes ago
Views: 155 • Comments: 0
Tweets: 6 • Rating: 1

researchok
French Philosopher Alain Badiou on the Real Expression of Love: 'If You Limit Yourself to Sexual Pleasure It's Narcissistic'
8 hours, 49 minutes ago
Views: 93 • Comments: 0
Tweets: 0 • Rating: 0

Haywood Jabloeme
Closing the Racial and Generational Divides
12 hours, 38 minutes ago
Views: 94 • Comments: 0
Tweets: 0 • Rating: 1

Aziz Poonawalla
I Speak for Myself: All-American
16 hours, 35 minutes ago
Views: 142 • Comments: 0
Tweets: 0 • Rating: 5

Mostly sane, most of the time.
So wake up and notice already
2 days, 12 hours ago
Views: 295 • Comments: 0
Tweets: 2 • Rating: 7

Daniel Ballard
Late Afternoon Light-Kalanchoe
3 days, 16 hours ago
Views: 271 • Comments: 0
Tweets: 0 • Rating: 5

Eclectic Infidel
City College of San Francisco Budget Update
3 days, 17 hours ago
Views: 324 • Comments: 0
Tweets: 0 • Rating: 1

MichaelJ
Apple TV Slated to Debut in December?
4 days, 23 hours ago
Views: 372 • Comments: 0
Tweets: 0 • Rating: 1

 Frank says:

People who think of videos as an art form are probably the same people who think Cabbage Patch Dolls are a revolutionary form of soft sculpture. -- Zappa on videos (obviously) from Viva Zappa - Biography