LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Desktop
User Name
Password
Linux - Desktop This forum is for the discussion of all Linux Software used in a desktop context.

Notices

Reply
 
Search this Thread
Old 02-09-2011, 02:59 PM   #1
deathsfriend99
Member
 
Registered: Nov 2007
Distribution: CentOS 6
Posts: 185

Rep: Reputation: 22
Slow system performance


This question really encompases allot of different toics, so I thought I'd throw it in here and hope something sticks.

I have 60+ client desktops all running CentOS 5.3 or higher. They are all pretty powerful machines (core2quads and corei7's) with between 4 and 8GB of RAM. The problem is they all run really slow. Frequent system stalls (2-3 secs of unresponsiveness) while running mundane things like emacs and firefox are very common. First I suspected video issues. I have tried both ATI and NVIDIA as well as their respective generic and proprietary drivers with no change.

I am serving all user profiles from an NFS/NIS server (quad core 8GB RAM). I am begining to wonder if there is a network bottleneck, or even how to begin troubleshooting that.

I'm really at a loss to figure out what the issue is, but it's becoming so bad, clients have been avoiding their desktops and have switched to using their personal (GASP) windows laptops due to the annoyance.

Any ideas where I could find start to find out what is bogging down my systems? Either internally or network?
 
Old 02-09-2011, 08:50 PM   #2
lugoteehalt
Senior Member
 
Registered: Sep 2003
Location: UK
Distribution: Debian
Posts: 1,215
Blog Entries: 2

Rep: Reputation: 49
Dunno but there are quite a few other threads about Centos being slow on this site, e.g. http://www.linuxquestions.org/questi...ed-why-634921/ . Hope some help.
 
Old 02-09-2011, 11:43 PM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,431

Rep: Reputation: 1066Reputation: 1066Reputation: 1066Reputation: 1066Reputation: 1066Reputation: 1066Reputation: 1066Reputation: 1066
Hmmm - Centos 5.3 might be too old, but maybe have a look at latencytop.
 
Old 02-10-2011, 10:29 AM   #4
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,916

Rep: Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777Reputation: 777
Quote:
Originally Posted by deathsfriend99 View Post
I have 60+ client desktops all running CentOS 5.3 or higher. They are all pretty powerful machines (core2quads and corei7's) with between 4 and 8GB of RAM.
Those ought to be respectably quick machines; so there is something wrong, but what?

Quote:
Originally Posted by deathsfriend99 View Post
Frequent system stalls (2-3 secs of unresponsiveness) while running mundane things like emacs and firefox are very common. First I suspected video issues. I have tried both ATI and NVIDIA as well as their respective generic and proprietary drivers with no change.
Good that you have checked out proprietary video, because that was a possibility.

Quote:
Originally Posted by deathsfriend99 View Post
I am serving all user profiles from an NFS/NIS server (quad core 8GB RAM). I am begining to wonder if there is a network bottleneck, or even how to begin troubleshooting that.
I would guess this is the most likely; perhaps with, eg, wireshark you could look at packets going from a client machine to the NFS/NIS server, and see what the time delay is.

The other question is whether the 2-3 seconds of unresponsiveness is the only kind of slowness that you have? In other words, if you do intense things that are purely local, do those seem fast?

What about grabbing files from NFS? Is that OK?

Please ensure that you have IPV6 turned off. Also, if the slowness is purely internet-related, you could also check that DNS name lookups are reasonably swift (and not, eg, trying a non-existant nameserver first, before going over to the one that actually does give an answer).

If I were to guess, I would guess that, for one reason or another, the systems are doing a lot of waiting (and that might be network or disk); maybe top and friends might show something interesting.
 
Old 02-15-2011, 03:10 PM   #5
deathsfriend99
Member
 
Registered: Nov 2007
Distribution: CentOS 6
Posts: 185

Original Poster
Rep: Reputation: 22
Thanks for the reply. I do believe it is network or NFS related. A system not mounting NFS has no performance issues. Once connected to our NIS/NFS, performance slows. All home directories mount on the NFS server. It does show considerable disk and network activity, but I would expect the throughput of a system like that to be able to handle it. It has plenty of RAM, 100Mbs Fullduplex, and SATA HD's.
Wireshark didn't show anything glaring, although I was amazed at the number of requests to and from the NFS machine. Not sure if that is normal.
gkrellm shows:
CPU average 10%
Disk average 5.5M
Eth0 average 2M
 
Old 02-15-2011, 06:08 PM   #6
fbsduser
Member
 
Registered: Oct 2009
Distribution: Hackintosh, SlackWare
Posts: 266

Rep: Reputation: 30
Is your server's SATA controller set as "IDE emulation" (or something like that, you check that in the BIOS setup)? If it is set like that you'll get a very low throughput (since it's essentially emulating an IDE port). To fix it you need to set it to "SATA" or "AHCI" which is the native mode and will yield the full throughput of your SATA controller/disks.
 
Old 02-16-2011, 09:00 PM   #7
DJ Shaji
Member
 
Registered: Dec 2004
Location: India
Distribution: Debian Squeeze, Fedora 17
Posts: 440
Blog Entries: 15

Rep: Reputation: 80
I'm so not not qualified to comment here, but have you tried

* using a custom kernel?
* using a different desktop environment?
* turning off cron jobs?
* Updating frequently used packages to their latest versions?

Are you sure the system stalls are random? I mean, generally the kernel can be caught up in disk io for a few seconds and the system may get stuck. You could try the "noasync" flag for mounting the root or other local filesystems.
 
Old 02-17-2011, 12:57 PM   #8
deathsfriend99
Member
 
Registered: Nov 2007
Distribution: CentOS 6
Posts: 185

Original Poster
Rep: Reputation: 22
Quote:
Originally Posted by fbsduser View Post
Is your server's SATA controller set as "IDE emulation" (or something like that, you check that in the BIOS setup)? If it is set like that you'll get a very low throughput (since it's essentially emulating an IDE port). To fix it you need to set it to "SATA" or "AHCI" which is the native mode and will yield the full throughput of your SATA controller/disks.
I don't recall. I haven't rebooted this machine in 6+months as it is the heart of the department. Is there a way to check that without going into the bios? All the drives are listed in /dev/sd# so I figured they were in sata mode.


Quote:
Originally Posted by DJ Shaji View Post
I'm so not not qualified to comment here, but have you tried

* using a custom kernel?
* using a different desktop environment?
* turning off cron jobs?
* Updating frequently used packages to their latest versions?

Are you sure the system stalls are random? I mean, generally the kernel can be caught up in disk io for a few seconds and the system may get stuck. You could try the "noasync" flag for mounting the root or other local filesystems.
These are options, but CentOS is a very stable and widely used distro. It's NFS capabilities are standard "out of the box" and shouldn't need customization. I may look into trying different mounting options though. That is a great idea.
 
Old 02-17-2011, 01:31 PM   #9
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,156

Rep: Reputation: 247Reputation: 247Reputation: 247
Hi


100mbit isn't very fast, it's only a fraction of the speed of a modern SATA disk. Even with one user, it's going to be the bottleneck.
 
Old 02-18-2011, 05:53 PM   #10
fbsduser
Member
 
Registered: Oct 2009
Distribution: Hackintosh, SlackWare
Posts: 266

Rep: Reputation: 30
Quote:
Originally Posted by deathsfriend99
I don't recall. I haven't rebooted this machine in 6+months as it is the heart of the department. Is there a way to check that without going into the bios? All the drives are listed in /dev/sd# so I figured they were in sata mode.
All drives will be listed as /dev/sd# regardless of what they`re connected to (IDE,SCSI,SATA), because of the way the kernel (more preciselly the libata library) handles disk connections. Essentially there`s AFAIK no way, other than going to the BIOS, of checking whether the controller is in native or legacy mode.
 
Old 02-25-2011, 02:34 PM   #11
arizonagroovejet
Senior Member
 
Registered: Jun 2005
Location: England
Distribution: SLED, openSUSE
Posts: 1,032

Rep: Reputation: 174Reputation: 174
Quote:
Originally Posted by deathsfriend99 View Post
100Mbs Fullduplex,
Really? Gigabit ethernet has been around for many years. Why do you only have 100 Mbit/s? That sounds like it's going to be a problem because you're going to get maybe 10MB/s read/write speeds if you're lucky and that will be shared between all your 60 machines.

I administer a bunch of machines that use home directories mounted via NFS. I don't know the exact specs of the server since someone else looks after it, but I do know that it has a networking that's a LOT faster than 100Mbit/s (I'd guess it's 10GbE) and the home directories on are on an Enterprise grade hardware raid array stuffed full of drives that are almost certainly spinning a lot faster than yours are, that's connected to the server via fibre optic. The desktop machines have 1000Mbit/s connections back to the nearest switch. If I run this with the working directory set to my NFS mounted home directory
Code:
$ dd if=/dev/zero of=foo bs=1024 count=1048576
I get a write speed of around 20MB/s.

Now look at the set up I've just described. Now look at what the server specs you say you have. Now look at mine again and consider how much better it is and that I'm getting 20MB/s which, let's face it, is slow when you compare to a local disk.

Try that command for yourself with the working directory set to a NFS share and then again with the working directory set to somewhere on the local disk. You may find the results interesting.



When you say
Quote:
I am serving all user profiles from an NFS/NIS server
what do you mean by 'user profiles'? 'user profiles' is a concept I always associate with Windows, where the concept of a 'home directory' doesn't really seem to exist. Are you mounting user's home directories from the NFS server, or something else?
 
Old 02-25-2011, 10:28 PM   #12
bluebox
Member
 
Registered: Jun 2004
Posts: 71

Rep: Reputation: 6
What about the simple things ...

What does ifconfig say? Suspicious errors or dropped packets?

Anything about "eth0 link down" in dmesg? Anything suspicious there? Esp on the server?

Does the NIC share its IRQ with the graphics?

60 clients are connected to ... what? Is this "what" simply running hot, maybe?

Set up a simple ftp server on the server and do some basic throughput/reliability tests to the clients.

Do all clients hang at once?

Is there some kind of traffic control? Else, this guy with the 4 GB BMP desktop picture of his spouse will eat up other peoples bandwidth. Have a look at "ntop" or similar.

Firefox hanging does not say much ... but emacs? What is emacs trying to do when it hangs? Saving something? Is it the X11-emacs or console emacs?

Review the stuff stored on the server. There's not much sense in storing Firefox cache, desktop themes and similar things remotely.

Does the server use swap?

You're not doing wireless, do you?
 
Old 02-26-2011, 12:31 AM   #13
deathsfriend99
Member
 
Registered: Nov 2007
Distribution: CentOS 6
Posts: 185

Original Poster
Rep: Reputation: 22
Quote:
Originally Posted by bluebox View Post
What does ifconfig say? Suspicious errors or dropped packets?
Nothing suspicious. No errors. no collisions, no dropped packets. nfsstat shows 3 retransmissions out of over 70,000,000 calls, so nothing odd there.
Quote:
Originally Posted by bluebox View Post
Anything about "eth0 link down" in dmesg? Anything suspicious there? Esp on the server?
No eth0 link down in dmesg or anything out of the ordinary. Just iptables messages. Not sure what Esp is.
Quote:
Originally Posted by bluebox View Post
Does the NIC share its IRQ with the graphics?
No.
Quote:
Originally Posted by bluebox View Post
60 clients are connected to ... what? Is this "what" simply running hot, maybe?
This is a Core2Quad machine. lm_sensors gives 37C for the cores. Not overly hot for this CPU.

Quote:
Originally Posted by bluebox View Post
Set up a simple ftp server on the server and do some basic throughput/reliability tests to the clients.
Havn't tried FTP but izone showed decent throughput.
Quote:
Originally Posted by bluebox View Post
Do all clients hang at once?
Not sure. It's only for a second or 2, and it's transient and random so it's not easy to diagnose.
Quote:
Originally Posted by bluebox View Post
Is there some kind of traffic control? Else, this guy with the 4 GB BMP desktop picture of his spouse will eat up other peoples bandwidth. Have a look at "ntop" or similar.
There is no traffic control. I have been unable to get ntop installed due to some dependency issues in CentOS. I'll keep looking.
Quote:
Originally Posted by bluebox View Post
Firefox hanging does not say much ... but emacs? What is emacs trying to do when it hangs? Saving something? Is it the X11-emacs or console emacs?
Just typing in emacs, it can hang for a few seconds then catch up and all your text appears. It is both X-11 and terminal emacs.
Quote:
Originally Posted by bluebox View Post
Review the stuff stored on the server. There's not much sense in storing Firefox cache, desktop themes and similar things remotely.
Haven't looked into this.
Quote:
Originally Posted by bluebox View Post
Does the server use swap?
Yes, 2X RAM.
Quote:
Originally Posted by bluebox View Post
You're not doing wireless, do you?
No.


One other thing, we are running a software RAID5, and sitting in the same room as the server, the drives are constantly running (ie: churning away). Maybe we should look into a hardware RAID? Is this more efficient?
 
Old 02-26-2011, 03:01 PM   #14
bluebox
Member
 
Registered: Jun 2004
Posts: 71

Rep: Reputation: 6
Quote:
Originally Posted by deathsfriend99 View Post
Not sure what Esp is.
Sorry, "esp" is a lazy abbreviation for "especially".


Quote:
Originally Posted by deathsfriend99 View Post
This is a Core2Quad machine. lm_sensors gives 37C for the cores. Not overly hot for this CPU.
So, your server is equipped with 60 NICs, directly serving 60 clients? No Switches inbetween?


Quote:
Originally Posted by deathsfriend99 View Post
Havn't tried FTP but izone showed decent throughput.
Izone? You mean iozone? This would be a filesystem benchmark, helpful only when run on the clients to benchmark NFS performance. Throughput is not your problem, but freezes. Does iozone show freezes?

Quote:
Originally Posted by deathsfriend99 View Post
It's only for a second or 2, and it's transient and random so it's not easy to diagnose.
Right. Linux usually throws some timeouts when filesystems or networks are hanging. But 2 seconds usually are not enough for a timeout. So, one way to diagnose your problem would be to put more stress on your network, with the intention to make things worse and finally get some explicit error messages.

Quote:
Originally Posted by deathsfriend99 View Post
Just typing in emacs, it can hang for a few seconds then catch up and all your text appears. It is both X-11 and terminal emacs.
This is strange. When "just typing", there should be no network activity that could make emacs hang due to hanging NFS. Even though your server could very well be the culprit, it's still possible that there is some hardware problem on the clients, maybe always present, bot more noticeable when running in NFS and increased network traffic. Especially when all clients are build from identical hardware.

Quote:
Originally Posted by deathsfriend99 View Post
Yes, 2X RAM.
I asked whether the server _uses_ swap, not whether there is swap. Swapping out harddisk content to harddisk again is a good prerequisite to slow down fileserving.

Quote:
Originally Posted by deathsfriend99 View Post
One other thing, we are running a software RAID5, and sitting in the same room as the server, the drives are constantly running (ie: churning away). Maybe we should look into a hardware RAID? Is this more efficient?
Your bottleneck most likely is the network, except the case you really have 60 100 MB NICs inside your server. Hardware raid will lower CPU usage and transfer stress on the chipset, but I wouldn't expect this to solve your freeze-problem.

Hardware raid will not stop your harddrives from churning.
 
Old 02-26-2011, 03:06 PM   #15
arizonagroovejet
Senior Member
 
Registered: Jun 2005
Location: England
Distribution: SLED, openSUSE
Posts: 1,032

Rep: Reputation: 174Reputation: 174
Quote:
Originally Posted by bluebox View Post
So, your server is equipped with 60 NICs,
Where do you get that the server has 60 NICS in it? That sounds like a rather implausible number of NICS to have in a server.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
slow performance lin_apps Linux - Newbie 3 10-02-2009 08:07 PM
Performance of CentOS 5.1 is slow...very slow! 90% mem used!! Why? Jazzmaster777 Linux - Server 5 04-13-2008 06:48 AM
Really Slow Performance in 12.0 ppr:kut Slackware 15 07-30-2007 09:30 AM
Slow performance!!! System using a lot of Physical Memory SLOW HELP! badgerbox76 Linux - Newbie 31 07-19-2007 11:23 AM
slow performance dominant Linux - Newbie 20 03-16-2004 10:56 AM


All times are GMT -5. The time now is 11:30 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration