I'm stumped.

I have two Linux boxes. They run the same application, and have done so admirably for a few years. Lately one has slowed way down. (It's about 20 times slower running my application)

The tortoise has 8 CPUs, Two Quad Core Xeon processors on a Supermicro MB. 8 GB of RAM and a RAID 5 disk array powered by a 3Ware 9550SX RAID card.

The hare has 8 CPU's. Four Dual core AMD processors on a Supermicro MB. 8 GB of RAM and a RAID 5 disk array powered by a 3Ware 9550SX RAID card.

Yesterday, I tried to run a job on the tortoise, and it ran for 20 hours. I copied all the data to the hare, and it ran in 1 hour.

The hare is running Centos 2.6.18-164.11.1.el5
The Tortoise is running CentOS 2.6.18-164.15.1.el5

There is nothing odd in /var/log/messages
I have NOT tried to run memtest. (I always have trouble
with that anyway, as I have multiple CPUs and more than 4GB RAM)
There are no messages in the error logs for the RAID cards.
Running a 'cp' command on a big file on the tortoise and watching iostat shows that I'm transferring at about 60 MB/sec. (This stays the same whether I copy between the RAID filesystem and a system disk, or any combination of those two. And it stays the same if I use my application to do the copying.)
It doesn't look like a disk problem.

Running mpstat shows that my application caused the CPUs to spend most of their time in IOWait.

vmstat does not show massive swapping (or any)

I can't explain this, or even how to figure out what's going on.
Is there any way it could be the network? Both machines are on the same network. They do not use the network for anything, it's just there. I've just noticed that over my many years of head banging over computers, is that when one computer on a network slows down, they all seem to. Even when the network isn't used. (I know that's impossible, but then if man were meant to fly God would have given him wings, or a stronger velocity fart expulsion.)

2+ hours and no reply ... time for the ignoramuses to try and help out ...

Have you used a network traffic monitor such as tcpdump to confirm that? IDK, but could IOWait time be waiting for network response? Alternatively, could IOWait time be waiting for file locks to clear?

4 days and 1 reply ... time for 2nd ignoramuses to join in ...

Maybe the 'tortoise' decided to take it slow and steady

Ok jokes aside,

Software side I cann't suggest much other than full reinstall (I don't think that is what you are looking for and it might not help, leave reinstall as lastresort).

chk for rootkits and things like that.

chk
Run the same 'cp' command on a big file on 'hare' and see transfer rate.


Hardware side chk:
-dust build up, faulty fans, overheating parts
-PSU(s)
-cache RAM in 3Ware 9550SX cards ok (if there any)
-remove 'tortoise' ram modules one at(or in pairs) a time and if that help
-remove 'tortoise' ram modules one at time and test it on different pc
-see if HDDs in 'tortoise' hotter than normal
-google and see if HDDs models in 'tortoise' have any known problems.