LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-30-2007, 11:50 AM   #1
MRosanbalm
LQ Newbie
 
Registered: Mar 2007
Location: Salem, Oregon USA
Distribution: Red Hat, Solaris
Posts: 6

Rep: Reputation: 0
Red Hat webserver slowing down


I've been assigned to discover why a web server is slowing down.

System is Linux FedoraCore 2 2.6.14. 2.6 gig proc, 1024 meg ram, It's a pretty vanilla system, hosting about 70 sites, most small a few big sites. They run MySql, php, various scripts. Of course the first suspect would be a runaway user script, and that will be verified, but for the sake of discussion, assume there are no user scripts running amuck. What system services should I check/verify.

My background: Couple years experience, comfortable at the CLI. System background: Runs for a day or so then starts the slowing down routine. A reboot always brings the speed back up. Haven't done any trouble shooting as of yet. Any suggestions would be very appreciated.

This is my first post, and I tried to make it lest verbose as possible but with enough information. Any thoughts regarding my post etiquette or regarding the server is very welcome.

Thanks,
73
Mike
 
Old 04-01-2007, 04:46 PM   #2
hob
Senior Member
 
Registered: Mar 2004
Location: Wales, UK
Distribution: Debian, Ubuntu
Posts: 1,075

Rep: Reputation: 45
It sounds like a memory leak. Unfortunately, these can occur in pretty much any C or C++ code, including supporting libraries or plugins. Your first weapon is simply "top". If it shows that Apache is the problem then ensure that you check the active modules - I've recently seen mod_python repeatedly kill a Web server by causing Apache processes to consume all of the RAM. In that case, switching the Python app to FastCGI and disabling the bad copy of mod_python got things working again within minutes.
 
Old 04-05-2007, 09:22 AM   #3
MRosanbalm
LQ Newbie
 
Registered: Mar 2007
Location: Salem, Oregon USA
Distribution: Red Hat, Solaris
Posts: 6

Original Poster
Rep: Reputation: 0
System slowing

Thanks Hob for the thoughts, I have been watching TOP for a while, and the interesting thing is even when the system slows down, TOP shows nothing out of the ordinary. I understand the difference between memory usage and Cache, so memory is okay. Also CPU usage is extremely low. It is the strangest thing, the whole thing just slows to a crawl, then reboot and magically everything is working fine again. Even TOP sends it's output slowly to my SSH session, but it reports all is well. Hmmmmmmm.....

Mike
 
Old 04-05-2007, 03:01 PM   #4
hob
Senior Member
 
Registered: Mar 2004
Location: Wales, UK
Distribution: Debian, Ubuntu
Posts: 1,075

Rep: Reputation: 45
You probably need to look at disk and network I/O next. Also check the available disk space to ensure that temporary files aren't consuming too much - as a rule of thumb, hard drives should have 10% free space to run well. The most worrying possibility is that the server has been hacked, and software has been installed that hides itself, but it is more likely that a process is progressively using up some resource.
 
Old 04-06-2007, 09:17 AM   #5
MRosanbalm
LQ Newbie
 
Registered: Mar 2007
Location: Salem, Oregon USA
Distribution: Red Hat, Solaris
Posts: 6

Original Poster
Rep: Reputation: 0
server slowing

Well, we're not a real big target, but the hack thing is worry sum. An interesting part is of the other servers, only two are acting this way. Each of the two host huge hotel rez. sites, but the respective tmp space usage seems to be in line. There is over two gig of swap space, do you think we should expand that? The machines have plenty of disk space. (80 gig) Here are the divisions...

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 521748 234052 261192 48% /
none 516696 0 516696 0% /dev/shm
/dev/sda7 65288364 41896336 23392028 65% /home
/dev/sda5 5114432 1725996 3388436 34% /usr
/dev/sda6 5114432 3383036 1731396 67% /var

That seems pretty normal...

The real confusing part is the way it slowly ramps up the slowing. It can take a day or two to slow down. I reboot and all is well again for a day or two. One of the fellow here wants to place a job in cron to reboot at 3am every morning. I would rather find the problem, but management wants the site up no matter what. My guess is the process gone amuck theory. I just need to figure out a way beyond top and ps -aux to watch/monitor that. What if I created a script which would grep the top process running every minute or so and toss it into a file for later examination? At any rate, thanks for your ear.

Mike
 
Old 04-06-2007, 10:26 AM   #6
digen
Member
 
Registered: Dec 2005
Location: India
Distribution: Ubuntu Feisty Fawn
Posts: 107

Rep: Reputation: 15
What is the output of the command given below when things are crawling ?

Quote:
# freem -mt
 
Old 04-07-2007, 12:08 PM   #7
MRosanbalm
LQ Newbie
 
Registered: Mar 2007
Location: Salem, Oregon USA
Distribution: Red Hat, Solaris
Posts: 6

Original Poster
Rep: Reputation: 0
server slowing

I don't see where "freem -mt" is a unix/linux command?!?!
would that be something unique to Ubuntu?
Did I miss something?

Mike,
 
Old 04-07-2007, 03:36 PM   #8
hob
Senior Member
 
Registered: Mar 2004
Location: Wales, UK
Distribution: Debian, Ubuntu
Posts: 1,075

Rep: Reputation: 45
Quote:
Originally Posted by MRosanbalm
There is over two gig of swap space, do you think we should expand that? The machines have plenty of disk space. (80 gig) Here are the divisions...

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 521748 234052 261192 48% /
none 516696 0 516696 0% /dev/shm
/dev/sda7 65288364 41896336 23392028 65% /home
/dev/sda5 5114432 1725996 3388436 34% /usr
/dev/sda6 5114432 3383036 1731396 67% /var

That seems pretty normal...

The real confusing part is the way it slowly ramps up the slowing. It can take a day or two to slow down. I reboot and all is well again for a day or two. One of the fellow here wants to place a job in cron to reboot at 3am every morning. I would rather find the problem, but management wants the site up no matter what. My guess is the process gone amuck theory. I just need to figure out a way beyond top and ps -aux to watch/monitor that. What if I created a script which would grep the top process running every minute or so and toss it into a file for later examination? At any rate, thanks for your ear.

Mike
The performance difference between memory and disks is so huge that you should really avoid swap - the absolute ideal is to keep the entire working dataset in memory.

Your df output looks fine, although it's worth looking in /tmp to see what's there. I mentioned network I/O as well as disks because I had a problem with a Windows server some time ago where the network card driver kept adding packets to one of it's queues until the network performance of the interface collapsed. The moral of the story being that the server itself was fine, it was just the networking that was failing...

If you have several Linux application servers then it's worth investing the time in installing some form of monitoring for them on a separate machine - there are a number of free products. That way you would be able to centrally log over time the kinds of info that top and other utilities give you, for all of the systems.
 
Old 04-08-2007, 12:18 PM   #9
MRosanbalm
LQ Newbie
 
Registered: Mar 2007
Location: Salem, Oregon USA
Distribution: Red Hat, Solaris
Posts: 6

Original Poster
Rep: Reputation: 0
server slowing

Hob, I am so glad you brought that up, I have several questions regarding monitoring. Quick background: I am currently enrolled in the Cisco networking academy at our local college, which is part of a baccalaureate in IS. I believe we should have spent more time working in monitoring. There was considerable time spent in IDS and IPS, but as far as I was concerned not nearly enough time in general performance monitoring.

So, my question doesn’t involve Linux, it involves Solaris, but actually I am sure the basic concepts would work with any flavor. I understand how to initiate logging from any given piece of equipment such as a Router, Pix, Server, etc. And on an x86 platform I can use Kiwi among many others. I am aware Solaris, (and Linux I’m sure) have log reception enabled by default, but where are the results kept? And which daemon is responsible.

I have trolled the net several times looking for answers, however I am afraid I am searching using the wrong terms. My prof’s seem “too busy” to concern themselves with such nonsensical notions of simple monitoring. <Grin> This task I am involved in is a part time jr. SysAdmin position for a hosting firm. I love this work because it is so diverse in nature, and the downside is it is so diverse in nature. <Grin>

These folks here at the firm are nice folks but spread a little thin. Monitoring I feel would be a great addition to the arsenal of tools they should have. And even though the slowing problem is a dire one and one demanding attention, without some form of monitoring we have to resort to logging in 24/7 and eyeballing things. So I am left deciding which fire to douse first. The Sr. Admin has his own fires to douse too. The hotel sites I previously mentioned are Really big and change daily, so that is where his time is spent, and I am delegated to the above mentioned issue(s). Okay, that’s my story. My intention wasn’t to talk your ear off, but I thought it helpful to know something about the background here.

Remember, like Dorothy said, there’s no place like 127.0.0.1

Mike
 
Old 04-08-2007, 01:08 PM   #10
hob
Senior Member
 
Registered: Mar 2004
Location: Wales, UK
Distribution: Debian, Ubuntu
Posts: 1,075

Rep: Reputation: 45
Quote:
Originally Posted by MRosanbalm
Hob, I am so glad you brought that up, I have several questions regarding monitoring. Quick background: I am currently enrolled in the Cisco networking academy at our local college, which is part of a baccalaureate in IS. I believe we should have spent more time working in monitoring. There was considerable time spent in IDS and IPS, but as far as I was concerned not nearly enough time in general performance monitoring.

So, my question doesn’t involve Linux, it involves Solaris, but actually I am sure the basic concepts would work with any flavor. I understand how to initiate logging from any given piece of equipment such as a Router, Pix, Server, etc. And on an x86 platform I can use Kiwi among many others. I am aware Solaris, (and Linux I’m sure) have log reception enabled by default, but where are the results kept? And which daemon is responsible.
There's a bit of a disjunct between the academic and commercial courses, and IT as it is actually practiced. AFAIKT it seems to be a general problem, so I guess admins just have to be prepared to do their own learning to fill the gaps.

Logging is handled by a syslog service, which writes to files in /var/log/. Most Linux systems uses software just called "syslog", but a couple of distros use "syslog-ng", or will do so. By default, syslog on Linux is configured not to accept log messages from other hosts - I'm not sure about Solaris. Regular syslog has weak security, so you have to take care to restrict which systems the log host will accept messages from. The network protocol is also called syslog.

This is completely separate from SNMP, which is the usual protocol for network-wide monitoring. A syslog arrangement passes and stores text messages, which may have any format the sending application likes, whilst SNMP is more structured. There have been a couple of attempts by vendor consortiums to replace SNMP, but the proposed successors haven't really worked out, although Windows uses a varient of one of them, under the name WMI (Windows Management Instrumentation).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Webserver - Red Hat Enterprise sseeker Linux - Newbie 12 09-05-2006 11:07 AM
Red Hat 9.0 Webserver not displaying Asian fonts AHjuncti Linux - Software 2 09-19-2005 04:34 PM
Red Hat 9 Webserver Crash64 Linux - Newbie 3 12-23-2004 04:16 AM
Red Hat 9 / running webserver but can't use /var/www/html/ novice22 Linux - Security 1 01-28-2004 07:09 PM
Red Hat Router/Webserver/Mailserver/File Server SixShooterz Linux - Networking 3 07-11-2003 04:13 PM


All times are GMT -5. The time now is 06:41 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration