Slow system performance
 

This question really encompases allot of different toics, so I thought I'd throw it in here and hope something sticks.

I have 60+ client desktops all running CentOS 5.3 or higher. They are all pretty powerful machines (core2quads and corei7's) with between 4 and 8GB of RAM. The problem is they all run really slow. Frequent system stalls (2-3 secs of unresponsiveness) while running mundane things like emacs and firefox are very common. First I suspected video issues. I have tried both ATI and NVIDIA as well as their respective generic and proprietary drivers with no change.

I am serving all user profiles from an NFS/NIS server (quad core 8GB RAM). I am begining to wonder if there is a network bottleneck, or even how to begin troubleshooting that.

I'm really at a loss to figure out what the issue is, but it's becoming so bad, clients have been avoiding their desktops and have switched to using their personal (GASP) windows laptops due to the annoyance.

Any ideas where I could find start to find out what is bogging down my systems? Either internally or network?

Dunno but there are quite a few other threads about Centos being slow on this site, e.g. http://www.linuxquestions.org/questi...ed-why-634921/ . Hope some help.:)

Hmmm - Centos 5.3 might be too old, but maybe have a look at latencytop.

Those ought to be respectably quick machines; so there is something wrong, but what?

Good that you have checked out proprietary video, because that was a possibility.

I would guess this is the most likely; perhaps with, eg, wireshark you could look at packets going from a client machine to the NFS/NIS server, and see what the time delay is.

The other question is whether the 2-3 seconds of unresponsiveness is the only kind of slowness that you have? In other words, if you do intense things that are purely local, do those seem fast?

What about grabbing files from NFS? Is that OK?

Please ensure that you have IPV6 turned off. Also, if the slowness is purely internet-related, you could also check that DNS name lookups are reasonably swift (and not, eg, trying a non-existant nameserver first, before going over to the one that actually does give an answer).

If I were to guess, I would guess that, for one reason or another, the systems are doing a lot of waiting (and that might be network or disk); maybe top and friends might show something interesting.

Thanks for the reply. I do believe it is network or NFS related. A system not mounting NFS has no performance issues. Once connected to our NIS/NFS, performance slows. All home directories mount on the NFS server. It does show considerable disk and network activity, but I would expect the throughput of a system like that to be able to handle it. It has plenty of RAM, 100Mbs Fullduplex, and SATA HD's.
Wireshark didn't show anything glaring, although I was amazed at the number of requests to and from the NFS machine. Not sure if that is normal.
gkrellm shows:
CPU average 10%
Disk average 5.5M
Eth0 average 2M

Is your server's SATA controller set as "IDE emulation" (or something like that, you check that in the BIOS setup)? If it is set like that you'll get a very low throughput (since it's essentially emulating an IDE port). To fix it you need to set it to "SATA" or "AHCI" which is the native mode and will yield the full throughput of your SATA controller/disks.

I'm so not not qualified to comment here, but have you tried

* using a custom kernel?
* using a different desktop environment?
* turning off cron jobs?
* Updating frequently used packages to their latest versions?

Are you sure the system stalls are random? I mean, generally the kernel can be caught up in disk io for a few seconds and the system may get stuck. You could try the "noasync" flag for mounting the root or other local filesystems.

I don't recall. I haven't rebooted this machine in 6+months as it is the heart of the department. Is there a way to check that without going into the bios? All the drives are listed in /dev/sd# so I figured they were in sata mode.


These are options, but CentOS is a very stable and widely used distro. It's NFS capabilities are standard "out of the box" and shouldn't need customization. I may look into trying different mounting options though. That is a great idea.

Hi


100mbit isn't very fast, it's only a fraction of the speed of a modern SATA disk. Even with one user, it's going to be the bottleneck.

All drives will be listed as /dev/sd# regardless of what they`re connected to (IDE,SCSI,SATA), because of the way the kernel (more preciselly the libata library) handles disk connections. Essentially there`s AFAIK no way, other than going to the BIOS, of checking whether the controller is in native or legacy mode.

Really? Gigabit ethernet has been around for many years. Why do you only have 100 Mbit/s? That sounds like it's going to be a problem because you're going to get maybe 10MB/s read/write speeds if you're lucky and that will be shared between all your 60 machines.

I administer a bunch of machines that use home directories mounted via NFS. I don't know the exact specs of the server since someone else looks after it, but I do know that it has a networking that's a LOT faster than 100Mbit/s (I'd guess it's 10GbE) and the home directories on are on an Enterprise grade hardware raid array stuffed full of drives that are almost certainly spinning a lot faster than yours are, that's connected to the server via fibre optic. The desktop machines have 1000Mbit/s connections back to the nearest switch. If I run this with the working directory set to my NFS mounted home directory
I get a write speed of around 20MB/s.

Now look at the set up I've just described. Now look at what the server specs you say you have. Now look at mine again and consider how much better it is and that I'm getting 20MB/s which, let's face it, is slow when you compare to a local disk.

Try that command for yourself with the working directory set to a NFS share and then again with the working directory set to somewhere on the local disk. You may find the results interesting.



When you say
what do you mean by 'user profiles'? 'user profiles' is a concept I always associate with Windows, where the concept of a 'home directory' doesn't really seem to exist. Are you mounting user's home directories from the NFS server, or something else?

What about the simple things ...

What does ifconfig say? Suspicious errors or dropped packets?

Anything about "eth0 link down" in dmesg? Anything suspicious there? Esp on the server?

Does the NIC share its IRQ with the graphics?

60 clients are connected to ... what? Is this "what" simply running hot, maybe?

Set up a simple ftp server on the server and do some basic throughput/reliability tests to the clients.

Do all clients hang at once?

Is there some kind of traffic control? Else, this guy with the 4 GB BMP desktop picture of his spouse will eat up other peoples bandwidth. Have a look at "ntop" or similar.

Firefox hanging does not say much ... but emacs? What is emacs trying to do when it hangs? Saving something? Is it the X11-emacs or console emacs?

Review the stuff stored on the server. There's not much sense in storing Firefox cache, desktop themes and similar things remotely.

Does the server use swap?

You're not doing wireless, do you?

Nothing suspicious. No errors. no collisions, no dropped packets. nfsstat shows 3 retransmissions out of over 70,000,000 calls, so nothing odd there.
No eth0 link down in dmesg or anything out of the ordinary. Just iptables messages. Not sure what Esp is.
No.
This is a Core2Quad machine. lm_sensors gives 37C for the cores. Not overly hot for this CPU.

Havn't tried FTP but izone showed decent throughput.
Not sure. It's only for a second or 2, and it's transient and random so it's not easy to diagnose.
There is no traffic control. I have been unable to get ntop installed due to some dependency issues in CentOS. I'll keep looking.
Just typing in emacs, it can hang for a few seconds then catch up and all your text appears. It is both X-11 and terminal emacs.
Haven't looked into this.
Yes, 2X RAM.
No.


One other thing, we are running a software RAID5, and sitting in the same room as the server, the drives are constantly running (ie: churning away). Maybe we should look into a hardware RAID? Is this more efficient?

Sorry, "esp" is a lazy abbreviation for "especially".


So, your server is equipped with 60 NICs, directly serving 60 clients? No Switches inbetween?


Izone? You mean iozone? This would be a filesystem benchmark, helpful only when run on the clients to benchmark NFS performance. Throughput is not your problem, but freezes. Does iozone show freezes?

Right. Linux usually throws some timeouts when filesystems or networks are hanging. But 2 seconds usually are not enough for a timeout. So, one way to diagnose your problem would be to put more stress on your network, with the intention to make things worse and finally get some explicit error messages.

This is strange. When "just typing", there should be no network activity that could make emacs hang due to hanging NFS. Even though your server could very well be the culprit, it's still possible that there is some hardware problem on the clients, maybe always present, bot more noticeable when running in NFS and increased network traffic. Especially when all clients are build from identical hardware.

I asked whether the server _uses_ swap, not whether there is swap. Swapping out harddisk content to harddisk again is a good prerequisite to slow down fileserving.

Your bottleneck most likely is the network, except the case you really have 60 100 MB NICs inside your server. Hardware raid will lower CPU usage and transfer stress on the chipset, but I wouldn't expect this to solve your freeze-problem.

Hardware raid will not stop your harddrives from churning.

Where do you get that the server has 60 NICS in it? That sounds like a rather implausible number of NICS to have in a server.