After poring over system logs, I’ve discovered another big drain on our system resources. At the bottom right of each post on the front page are links to versions of the individual post with and without comments. Sometimes people link to the pages with comments, even though they’re only interested in the post itself; and since pages with comments are often much larger than pages without, this is a lot of unnecessary work for the LGF server.

To make it worse, there are web robots out there that ignore the standard ways of blocking them, and will try to crawl through every post at LGF, including reloading pages with comments over and over. Lots and lots of wasted work for our server, slowing down the site for everyone.

So to address this, I’ve implemented some code that checks the referrer information for each page request, and if the referrer is not littlegreenfootballs.com (or if there’s no referrer), the request is redirected to the cached version of the page without comments. Since I installed this code last night, it has already saved more than half a gigabyte of data.

Most people won’t ever notice this; if you haven’t changed your browser settings or installed some kind of firewall that blocks referrers, everything will continue to work as it always has.

However, I’ve received a couple of emails from people who say they’re now unable to access comments pages at all. The most likely cause is that your browser has been set to not send referrer information; if it’s happening to you, check your browser preferences and make sure you haven’t disabled the sending of referrer info.

UPDATE at 3/19/07 9:16:26 am:

Here’s how to make sure Firefox is set up to send referrer information:

1. In the address bar of your Firefox browser type: about:config and hit Enter.
2. In the Filter input field type: network.http.sendRefererHeader.
3. The value for this preference should be 2. If it isn’t, double click on the preference name.
4. Change the value to 2 and hit OK.

UPDATE at 3/19/07 9:18:58 am:

Also, Norton Internet Security and Personal Firewall may block the sending of referrer info by default. Here is information on enabling referrers if you use either of these products: Passing referrer information to specific Web pages in Norton Internet Security and Norton Personal Firewall.

UPDATE at 3/19/07 10:09:06 am:

Firefox users should also check out this extension that lets you manage how your referrer information is sent: RefControl.

UPDATE at 3/19/07 2:42:19 pm:

LGF reader Wicksy made a good suggestion: to set a session variable when a person is redirected to the ‘no comments’ page, so that any further requests for comment threads will be allowed to go through, during that session.

That means that if you’re a live human, you’ll only be redirected once. (Robots usually don’t accept session cookies.) After that you can go in and out of comments at will.