Little Green Footballs - Tech Note: The New Web Server's Here

Show top rated links | LinkViewer

Tech Note: The New Web Server's Here

Science | Tue, Nov 18, 2008 at 4:05:12 pm PST

Well, not quite here, but it’s set up and ready to migrate everything over from the current one, the one that’s serving the page you’re reading right now.

The old one is a dual core Xeon 3.0, the new one is a quad core Xeon 3.4 with hyperthreading, so the processor will be considerably more powerful, which should help prevent hamster fatigue and class war.

We’re not sure exactly when the changeover will happen, but there may be some downtime involved, so we’ll try to schedule it for a bit later when traffic tapers off somewhat.

Hit the tip jar!

refresh | ► spinoff links (2)

Tags: Tech Note, Maintenance, Web Server

76 comments

Comments are open and unmoderated, and do not necessarily reflect the views of Little Green Footballs.
Obscene, abusive, silly, or annoying remarks may be deleted, but the fact that particular comments remain on the site in no way constitutes an endorsement of their views by Little Green Footballs.
Posts that contain phone numbers, street addresses, email addresses or other personal information will also be deleted, as will posts that consist only of a variation on the word, "First!"
Comments that advocate violence will be cause for immediate banning with no appeal.
Disagreement and debate are welcome, but insults and abuse are not, and may cause your account to be blocked.
REMEMBER: posting comments at LGF is a privilege, not a right. Abuse that privilege, and your account will be blocked.

Hide comments | Jump to bottom

nobamadada Tue, Nov 18, 2008 4:06:33pm

Ahh quad core! Thank you Charles!

Russkilitlover Tue, Nov 18, 2008 4:07:04pm

Down time? Oh, NOES! Please give me advance warning so I can prepare myself with a sustaining mantra until LGF is back....."Find a happy place, find a happy place!"

tommygum Tue, Nov 18, 2008 4:07:42pm

All this tech talk is , for me anyway, casting pearls before swine.

nikis-knight Tue, Nov 18, 2008 4:08:53pm

Hmmm, let's see if I can post on this thread, after numerous spinning circles of doom over the last half hour on the previous...

Joo-LiZ Tue, Nov 18, 2008 4:09:08pm

Sounds sweet Charles! Looking forward to it!

ggt Tue, Nov 18, 2008 4:09:10pm

The new Server is here! The new Server is here! The new Server is here!

Oh happy day.

SWPaul Tue, Nov 18, 2008 4:09:51pm

Ooooooh! Sounds sexy!

Dar ul Harb Tue, Nov 18, 2008 4:10:10pm

So, does each thread with more than 500 comments get its own processor?

Stand by for ludicrous speed!

Killian Bundy Tue, Nov 18, 2008 4:10:16pm

The apology Bush owes Dan Rather

What is really in order is some sort of pardon and apology to Dan Rather, who CBS's cowardly management squeezed from 60 Minutes for telling the truth about Bush's war record. Rather's suit against them, with its accompanying subpoenas, has now revealed that in their eagerness to throw a sacrificial victim to the swiftboating bloggers with their escorting media sharks, CBS management actually considered such paragons of journalistic objectivity as Ann Coulter, Rush Limbaugh, William Kristol, William Safire and William Buckley for the investigating panel. Their concern was to counter the "reputation" of the network's liberal bias.

/hope this goes to trial

CharlieBravo Tue, Nov 18, 2008 4:12:02pm

Hamster holiday if only for a few hours... and well deserved.

notutopia Tue, Nov 18, 2008 4:12:06pm

Sweeeet! Thanks Charles!

callahan23 Tue, Nov 18, 2008 4:12:21pm

How can we ever repay you Charles for you being there.
Sticking to your principles, opposing radicalism of all forms and now a NEW SERVER!

Joo-LiZ Tue, Nov 18, 2008 4:12:35pm

re: #9 Killian Bundy

The apology Bush owes Dan Rather

/hope this goes to trial

Unbelievable.

Ditto what you said.

Kragar (proud to be kafir) Tue, Nov 18, 2008 4:12:56pm

re: #6 ggt

The new Server is here! The new Server is here! The new Server is here!

"Page 73 - Johnson, Navin R.! I'm somebody now! Millions of people look at this book everyday! This is the kind of spontaneous publicity - your name in print - that makes people. I'm in print! Things are going to start happening to me now."

/to mix quotes

debutaunt Tue, Nov 18, 2008 4:13:43pm

Wow - I'm sure it will increase my typing speed - I'm giddy!

6pat6 Tue, Nov 18, 2008 4:14:11pm

There has not yet been a hamster that can take the stress of what Lizards can inflict upon then.

Bring that quad core machine and all of its fresh hamsters on! We wore the dual-core out; the new one is just two more cores!

So, do the old hamsters now retire to the OHRH (Old Hamster Retirement Home), or do they get to be food for the Lizards?

Karridine Tue, Nov 18, 2008 4:14:40pm

Charles, how much RAM does main-server use? SRAM or DRAM?

And what cacheing scheme, please?

stuiec Tue, Nov 18, 2008 4:15:59pm

re: #2 Russkilitlover

Down time? Oh, NOES! Please give me advance warning so I can prepare myself with a sustaining mantra until LGF is back....."Find a happy place, find a happy place!"

Why don't you take a nice big dose of Metamucil now... and then when the site downtime occurs, you'll be able to kill two birds with one stone (so to speak)?

tommygum Tue, Nov 18, 2008 4:16:16pm

re: #15 debutaunt

Wow - I'm sure it will increase my typing speed - I'm giddy!

Amphetamines couldn't increase my typing speed.

Walter L. Newton Tue, Nov 18, 2008 4:16:37pm

re: #8 Dar ul Harb

So, does each thread with more than 500 comments get its own processor? Stand by for ludicrous speed!

Charles can probably go into more detail about this than myself, but in short, having a new server with a quad core processor will allow LGF to answer many more "requests" for pages/comment at the same time.

If you are having trouble loading comments after 400-500 comments, the problem is on your side.

I am using a six year old 3.0ghz speed computer, 1gig memory, hyperthreaded, and running XP.

I had speed problems (and lockups) after 400-500 comments. I was using IE7 browser.

I switched to Safari, no problems anymore.

Who Watches the Watchmen? Tue, Nov 18, 2008 4:17:03pm

STOP CLASS WAR!
GO QUAD CORE!

Bubblehead II Tue, Nov 18, 2008 4:17:47pm

We’re not sure exactly when the changeover will happen, but there may be some downtime involved, so we’ll try to schedule it for a bit later when traffic tapers off somewhat.

And that will be exactly when?

vxbush Tue, Nov 18, 2008 4:17:47pm

Maybe we should send in some cache to give Charles an extra 8 GB of RAM. If this is the machine I think it is, he can go up to 32 GB of RAM, right Charles?

Moe Katz Tue, Nov 18, 2008 4:18:30pm

re: #22 Bubblehead II

We’re not sure exactly when the changeover will happen, but there may be some downtime involved, so we’ll try to schedule it for a bit later when traffic tapers off somewhat.

And that will be exactly when?

When traffic tapers off. Your civic duty is to log off NOW.

6pat6 Tue, Nov 18, 2008 4:18:51pm

I almost feel faster, and The Fast One is not even up yeat!

scrad Tue, Nov 18, 2008 4:19:07pm

Actually, it would be interesting if you would post or put up a page showing how your network and software architecture is set up. Process flow, where's the database, what resources are on each servers like RAM, cache, disk speed.

Who knows, maybe some of us tech lizards might have some interesting suggestions or we'll learn something from your architecture...

Walter L. Newton Tue, Nov 18, 2008 4:20:26pm

re: #26 scrad

Actually, it would be interesting if you would post or put up a page showing how your network and software architecture is set up. Process flow, where's the database, what resources are on each servers like RAM, cache, disk speed.

Who knows, maybe some of us tech lizards might have some interesting suggestions or we'll learn something from your architecture...

:)

Yea, and maybe some hackers would like to know. Bad idea.

A Kiwi Infidel Tue, Nov 18, 2008 4:20:27pm

From 32 hamsters to 128, ah the hamster food budget has just gone out the window.

wrenchwench Tue, Nov 18, 2008 4:25:32pm

with hyperthreading

Oh, my.

Muadib Tue, Nov 18, 2008 4:25:56pm

In this case, more power is a good thing.

realwest Tue, Nov 18, 2008 4:26:59pm

Well as everyone knows, I'm the "go to guy" on LGF for tech problems and I must say I'm impressed Charles. Very impressed.
Uh, is it true that the new server's name is Jeeves (No, not that one, Jeeves as in Jeeves the Butler! sheesh!).

ghost707 Tue, Nov 18, 2008 4:31:40pm

The old one is a dual core Xeon 3.0, the new one is a quad core Xeon 3.4 with hyperthreading, so the processor will be considerably more powerful, which should help prevent hamster fatigue and class war.

LOL.

tommygum Tue, Nov 18, 2008 4:34:10pm

re: #28 A Kiwi Infidel

From 32 hamsters to 128, ah the hamster food budget has just gone out the window.

Feed them guv'mint cheese.

unclassifiable Tue, Nov 18, 2008 4:34:25pm

AHHHHHHHHHHHHHHHHH!

Something we can agree on!

debutaunt Tue, Nov 18, 2008 4:37:09pm

re: #32 ghost707

hahahahahahhahhaa

Walter L. Newton Tue, Nov 18, 2008 4:37:17pm

re: #31 realwest

Well as everyone knows, I'm the "go to guy" on LGF for tech problems and I must say I'm impressed Charles. Very impressed.
Uh, is it true that the new server's name is Jeeves (No, not that one, Jeeves as in Jeeves the Butler! sheesh!).

It appears you were the "go after" guy on the Infidel Blogger today. I thought I would be the hit of the day after Spencer suggested that I was libeling him, but you won, hands down.

scrad Tue, Nov 18, 2008 4:38:27pm

re: #27 Walter L. Newton

Yea, and maybe some hackers would like to know. Bad idea.

Wrong. Process flow diagrams do not expose code or server addresses. Knowing how much ram is on a computer, what kind of CPU and how the database and other processes, such as the web server, are distributed give you ideas on how you can IMPROVE the architecture.

'Hackers' will just throw all the denial of service attacks they want at the servers without needing to know (or care) what the internal architecture is.

That said, anybody in the applications / server realm would know that the easiest way to whack a site that depends on a mysql database, as most do, is to send lots of requests that will cause a non cacheable database query or one that causes a record to be updated or added.

The former means you force the db to work harder, the latter causes any indexes in MySQL to be flushed from the cache. Oracle doesn't have these problems but it's not free either.

You can put the most powerful web server in the world online, but if it's waiting for the database it's only as fast as the weakest link.

I have no idea how this site is architected, and I respect Charles engineering abilities. I just would be curious to see a 'sanitized' architecture overview.

I defend sites from this kind of stuff all the time. Some of the systems I've designed are running at AOLs datacenters.

Quilly Mammoth Tue, Nov 18, 2008 4:38:30pm

Charles, You ought to have them name the new server Stinky.

loflyer Tue, Nov 18, 2008 4:42:06pm

Good luck Charles, hope it works with out too many "issues". Some times this stuff doesn't work as advertised, and you get to back out, regroup and try again....

Charles Tue, Nov 18, 2008 4:44:52pm

re: #37 scrad

Two dedicated servers, one for web, one for DB. The bottleneck is definitely the processing speed of the web server. System load stays pretty low on the DB server, it's the web server and Apache that struggles to keep up with peak traffic.

mikeytrix Tue, Nov 18, 2008 4:47:44pm

All your hamsters are belong to us!

Who Watches the Watchmen? Tue, Nov 18, 2008 4:49:32pm

re: #9 Killian Bundy

What a schmuck.

Dr. Shalit Tue, Nov 18, 2008 4:50:11pm

re: #40 Charles

Two dedicated servers, one for web, one for DB. The bottleneck is definitely the processing speed of the web server. System load stays pretty low on the DB server, it's the web server and Apache that struggles to keep up with peak traffic.

Charles -

Everybody's Waitin' for the Weekend... Especially STINKY and the Hamsters.

-S-

Ghost707 Tue, Nov 18, 2008 4:51:06pm

Hamsters on JOLT cola!

Dr. Shalit Tue, Nov 18, 2008 4:52:39pm

re: #44 Ghost707

Hamsters on JOLT cola!

Ghost -

More like a better "Gear Ratio" for the Hamsters.

-S-

fredo malmstein Tue, Nov 18, 2008 4:59:18pm

Charles:

Are you sticking with Apple? Is it an xServe?

scrad Tue, Nov 18, 2008 5:01:51pm

re: #40 Charles

Two dedicated servers, one for web, one for DB. The bottleneck is definitely the processing speed of the web server. System load stays pretty low on the DB server, it's the web server and Apache that struggles to keep up with peak traffic.

I figured that was the case. If your setup is typical then each of your apache processes is 1-2 megs of ram with php running.

I've always wondered why you don't serve your static content with LiteHttpd? It's small and fast where Apache, being the 'swiss army knife' of servers has a lot of overhead. Put it on a second IP as 'static.littlegreenfootballs.com' or something like that.

And your URL structure implies that this is executing a php script:

/weblog/LizardLounge.gif

I'm sure you'll get a great improvement with your new server. Perhaps combining it with some minor software chages, like using LIteHttpd or pruning out of php the modules you are not using could add some icing to the cake.

Regardless Charles, you're one of the few who are capable of engineering a sophisticated system like this and I respect that. I especially like all the Ajax work you've done.

Charles Tue, Nov 18, 2008 5:07:12pm

re: #47 scrad

I also have the eAccelerator PHP compiler cache installed, which helps a lot. And I've done quite a bit of tweaking of the Apache and MySQL config files.

I have considered LiteHttpd too, and plan to investigate it further.

GeeWiz Tue, Nov 18, 2008 5:24:05pm

re: #6 ggt
Oh Happy Day

sngnsgt Tue, Nov 18, 2008 5:26:28pm

Better, stronger, faster.

psyop Tue, Nov 18, 2008 5:36:49pm

Those poor, old hamsters...

Charles, is PETA aware of their working conditions?

stevieray Tue, Nov 18, 2008 5:45:48pm

Good stuff!

And on a personal techie note, I just de-bricked my wireless router after a two month hibernation, and successfully installed dd.wrt on it!

/more dumb luck than skill... but still... yay!

gwillie Tue, Nov 18, 2008 5:51:19pm

Sounds like the hamsters are not getting a brake, just an extra wheel.

HoosierHoops Tue, Nov 18, 2008 6:03:26pm

re: #52 stevieray

Good stuff!

And on a personal techie note, I just de-bricked my wireless router after a two month hibernation, and successfully installed dd.wrt on it!

/more dumb luck than skill... but still... yay!

Isn't great that wireless standards are so simple and standardized?
*yea I'm gazing at you Microsoft*

stevieray Tue, Nov 18, 2008 6:08:44pm

re: #54 HoosierHoops

"Simple" and "standardized" don't seem to exist in the computer world... and wireless routers are confounded infernal satan machines to begin with. Why do they have so many %@*%# different addresses? Argh!

victor_yugo Tue, Nov 18, 2008 6:36:35pm

re: #48 Charles

I also have the eAccelerator PHP compiler cache installed, which helps a lot. And I've done quite a bit of tweaking of the Apache and MySQL config files.

I have considered LiteHttpd too, and plan to investigate it further.

LigHTTPd would be excellent for serving static content, especially images and JavaScripts. IIRC, Yahoo uses it for that purpose.

Just to be a fly in the ointment: HyperThreading isn't all Intel cracks it up to be. A single core with HyperThreading may be able to "think" about two things at once, but what one thing gains in speed, the other loses. There is still just one instruction stream passing through just one fetch-decode-execute-retire; each instruction is merely tagged with its context.

That said, going multi-core should help things a lot. And since you're running on Linux, you should be able to use per-CPU scheduling sets to keep similar tasks physically close to one another. Like, say, all kernel threads on Core 0, and let everything else fly on Cores 1, 2, and 3. YMMV, but I'm sure you know what kind of load you need to test it with.

Sloppy Tue, Nov 18, 2008 6:49:43pm

If I may be allowed to make a small correction, Jeeves was not a butler. He was a valet, or "gentleman's gentleman."

Sharmuta Tue, Nov 18, 2008 7:47:23pm

Ooooo! It's like Christmas came early!

scrad Tue, Nov 18, 2008 8:29:14pm

re: #56 victor_yugo

Just to be a fly in the ointment: HyperThreading isn't all Intel cracks it up to be. A single core with HyperThreading may be able to "think" about two things at once, but what one thing gains in speed, the other loses. There is still just one instruction stream passing through just one fetch-decode-execute-retire; each instruction is merely tagged with its context.

Yes, there were several studies don on Linux servers and windows where they were able to identify weaknesses in hyper threading.

Your description was right on.

Multi cores is better, but without some really large cache memory it still gets stuck in ram access.

The bottom line: reduce the memory footprint and the cpu footprint of your web services. minimize the number of context switches required to serve content.

Apache is a pig. It's a 'switch army knife'. Use it with caution.

LiteHttpd is optimized to do what it does: server static content.

and still, Charles has done an excellent job on this site.

suboptimal Tue, Nov 18, 2008 8:51:26pm

The parts for the dryer have shipped, the vent pipe is unclogged, my finger stopped bleeding, _and_ LGF is getting faster?

Best Tuesday ever!

iChef Tue, Nov 18, 2008 9:08:16pm

Charles,
You need to appoint an "Transition Team" for the "Office of Server-Elect" ...

Long ago in my dotcom days the company I worked for bought a program that put your whole SQL database into memory making super fast ... but it's probably an old idea now.

eeyore Tue, Nov 18, 2008 10:02:20pm

Glad the hampsters will have better working conditions. But I'm afraid The One has smilingly offered us at least four years of enthusiastic class warfare, no extra charge!

Graystoke Tue, Nov 18, 2008 10:31:05pm

Charles, why don't you use both servers and load balance them?

JamesS Tue, Nov 18, 2008 11:01:56pm

At last: not just change, but change for the better! Good luck, little hamsters: you have a tough job ahead of you...

victor_yugo Wed, Nov 19, 2008 12:06:49pm

re: #17 Karridine