Robots Crawling All Over Us

Charles Johnsonfollow me on twitter
Wed Aug 13, 2008 at 3:53 pm PDT • Views: 185

In the past few hours, I’ve discovered and blocked at least six web crawlers trying to go through the site and download everything as quickly as possible, in violation of the rules in our robots.txt file. Two of them seemed to be focusing on the posts related to the Georgia crisis.

Pretty unusual activity for a Wednesday afternoon.

22
Advertisement

220 comments

^ back to top ^

Name:

Pass:

Register Forgot Your Password? Account Settings Re-send Confirmation (To log in, cookies must be enabled in your browser!)

Turn off ads by subscribing!
For about 33 cents a day, our subscription option turns off all advertisements at LGF!
Read more...


► LGF Headlines

  • Loading...

► Tweeted Articles

  • Loading...

► Tweeted Pages

  • Loading...

► Top 10 Comments

  • Loading...

► Bottom Comments

  • Loading...

► Recent Comments

  • Loading...

► Tools/Info

► LGF Hits

► Resources

► Never Forget

► Statistics

► Tag Cloud

► Contact

You must have Javascript enabled to use the contact form.
 Your email:

Subject:

Message:


Messages may be published in our weblog, unless you request otherwise.
Tech Note:
Using the Contact Form
lgf cookbook 2
Reality Bites!
LGF Cookbook Volume 2

Partners
888-casino.eu Addiction Treatment Alcohol Treatment Alternative Medicine Dry Erase Board, White Boards, Whiteboards Fat Burners Front Row King Tickets Fundraisers Online Casino Sportsbook Tempurpedic Mattress Comparison Advertise here
More Partners

Compare Electricity Prices in your area. Texas Electricity is deregulated; you have the right to choose Texas Electric Rates from among many Texas Electric Companies.

An inner strongbox of shallow shibboleths.

TwitterFacebook
LGF Pages
Recent Pages

Ms. MacIceweasel
Half of Conservatives Are Now Against Breast Cancer Screenings
38 minutes ago
Views: 16 • Comments: 0
Tweets: 0 • Rating: 0

Randall Gross
Trial opens 2 years after feds break up Christian militia
1 hour, 9 minutes ago
Views: 47 • Comments: 0
Tweets: 0 • Rating: 0

Randall Gross
Obama's budget headed to Congress
1 hour, 24 minutes ago
Views: 56 • Comments: 0
Tweets: 0 • Rating: 0

researchok
Dreyfus and the Birth of Intellectual Protest
4 hours, 12 minutes ago
Views: 56 • Comments: 0
Tweets: 0 • Rating: 1

researchok
The Man Who Wouldn't Die: New York's Most Fantastic Murder
4 hours, 15 minutes ago
Views: 82 • Comments: 0
Tweets: 0 • Rating: 3

researchok
Mitt's Spin
4 hours, 17 minutes ago
Views: 55 • Comments: 0
Tweets: 0 • Rating: 0

researchok
Strategy
4 hours, 18 minutes ago
Views: 50 • Comments: 0
Tweets: 0 • Rating: 0

researchok
Gender Wage Gap Skewed by Survey Flaws - Miller-McCune
4 hours, 19 minutes ago
Views: 56 • Comments: 0
Tweets: 0 • Rating: 1

researchok
Why Obama Would Be Glad if the Culture War Is a Major Election Issue
4 hours, 19 minutes ago
Views: 75 • Comments: 0
Tweets: 0 • Rating: 1

researchok
Is This the End of Wall Street as They Knew It?
4 hours, 19 minutes ago
Views: 63 • Comments: 0
Tweets: 0 • Rating: 1

 Frank says:

Kid's heads are filled with so many nonfacts that when they get out of school they're totally unprepared to do anything. They can't read, they can't write, they can't think. Talk about child abuse. The U.S. school system as a whole qualifies. -- Discussing the state of the education system in America - Playboy magazine, April 1993.