I've been assigned to discover why a web server is slowing down.

System is Linux FedoraCore 2 2.6.14. 2.6 gig proc, 1024 meg ram, It's a pretty vanilla system, hosting about 70 sites, most small a few big sites. They run MySql, php, various scripts. Of course the first suspect would be a runaway user script, and that will be verified, but for the sake of discussion, assume there are no user scripts running amuck. What system services should I check/verify.

My background: Couple years experience, comfortable at the CLI. System background: Runs for a day or so then starts the slowing down routine. A reboot always brings the speed back up. Haven't done any trouble shooting as of yet. Any suggestions would be very appreciated.

This is my first post, and I tried to make it lest verbose as possible but with enough information. Any thoughts regarding my post etiquette or regarding the server is very welcome.

Thanks,
73
Mike

It sounds like a memory leak. Unfortunately, these can occur in pretty much any C or C++ code, including supporting libraries or plugins. Your first weapon is simply "top". If it shows that Apache is the problem then ensure that you check the active modules - I've recently seen mod_python repeatedly kill a Web server by causing Apache processes to consume all of the RAM. In that case, switching the Python app to FastCGI and disabling the bad copy of mod_python got things working again within minutes.

Thanks Hob for the thoughts, I have been watching TOP for a while, and the interesting thing is even when the system slows down, TOP shows nothing out of the ordinary. I understand the difference between memory usage and Cache, so memory is okay. Also CPU usage is extremely low. It is the strangest thing, the whole thing just slows to a crawl, then reboot and magically everything is working fine again. Even TOP sends it's output slowly to my SSH session, but it reports all is well. Hmmmmmmm.....

Mike

You probably need to look at disk and network I/O next. Also check the available disk space to ensure that temporary files aren't consuming too much - as a rule of thumb, hard drives should have 10% free space to run well. The most worrying possibility is that the server has been hacked, and software has been installed that hides itself, but it is more likely that a process is progressively using up some resource.

Well, we're not a real big target, but the hack thing is worry sum. An interesting part is of the other servers, only two are acting this way. Each of the two host huge hotel rez. sites, but the respective tmp space usage seems to be in line. There is over two gig of swap space, do you think we should expand that? The machines have plenty of disk space. (80 gig) Here are the divisions...

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 521748 234052 261192 48% /
none 516696 0 516696 0% /dev/shm
/dev/sda7 65288364 41896336 23392028 65% /home
/dev/sda5 5114432 1725996 3388436 34% /usr
/dev/sda6 5114432 3383036 1731396 67% /var

That seems pretty normal...

The real confusing part is the way it slowly ramps up the slowing. It can take a day or two to slow down. I reboot and all is well again for a day or two. One of the fellow here wants to place a job in cron to reboot at 3am every morning. I would rather find the problem, but management wants the site up no matter what. My guess is the process gone amuck theory. I just need to figure out a way beyond top and ps -aux to watch/monitor that. What if I created a script which would grep the top process running every minute or so and toss it into a file for later examination? At any rate, thanks for your ear.

Mike

What is the output of the command given below when things are crawling ?

I don't see where "freem -mt" is a unix/linux command?!?!
would that be something unique to Ubuntu?
Did I miss something?

Mike,

The performance difference between memory and disks is so huge that you should really avoid swap - the absolute ideal is to keep the entire working dataset in memory.

Your df output looks fine, although it's worth looking in /tmp to see what's there. I mentioned network I/O as well as disks because I had a problem with a Windows server some time ago where the network card driver kept adding packets to one of it's queues until the network performance of the interface collapsed. The moral of the story being that the server itself was fine, it was just the networking that was failing...

If you have several Linux application servers then it's worth investing the time in installing some form of monitoring for them on a separate machine - there are a number of free products. That way you would be able to centrally log over time the kinds of info that top and other utilities give you, for all of the systems.

Hob, I am so glad you brought that up, I have several questions regarding monitoring. Quick background: I am currently enrolled in the Cisco networking academy at our local college, which is part of a baccalaureate in IS. I believe we should have spent more time working in monitoring. There was considerable time spent in IDS and IPS, but as far as I was concerned not nearly enough time in general performance monitoring.

So, my question doesn’t involve Linux, it involves Solaris, but actually I am sure the basic concepts would work with any flavor. I understand how to initiate logging from any given piece of equipment such as a Router, Pix, Server, etc. And on an x86 platform I can use Kiwi among many others. I am aware Solaris, (and Linux I’m sure) have log reception enabled by default, but where are the results kept? And which daemon is responsible.

I have trolled the net several times looking for answers, however I am afraid I am searching using the wrong terms. My prof’s seem “too busy” to concern themselves with such nonsensical notions of simple monitoring. <Grin> This task I am involved in is a part time jr. SysAdmin position for a hosting firm. I love this work because it is so diverse in nature, and the downside is it is so diverse in nature. <Grin>

These folks here at the firm are nice folks but spread a little thin. Monitoring I feel would be a great addition to the arsenal of tools they should have. And even though the slowing problem is a dire one and one demanding attention, without some form of monitoring we have to resort to logging in 24/7 and eyeballing things. So I am left deciding which fire to douse first. The Sr. Admin has his own fires to douse too. The hotel sites I previously mentioned are Really big and change daily, so that is where his time is spent, and I am delegated to the above mentioned issue(s). Okay, that’s my story. My intention wasn’t to talk your ear off, but I thought it helpful to know something about the background here.

Remember, like Dorothy said, there’s no place like 127.0.0.1

Mike

There's a bit of a disjunct between the academic and commercial courses, and IT as it is actually practiced. AFAIKT it seems to be a general problem, so I guess admins just have to be prepared to do their own learning to fill the gaps.

Logging is handled by a syslog service, which writes to files in /var/log/. Most Linux systems uses software just called "syslog", but a couple of distros use "syslog-ng", or will do so. By default, syslog on Linux is configured not to accept log messages from other hosts - I'm not sure about Solaris. Regular syslog has weak security, so you have to take care to restrict which systems the log host will accept messages from. The network protocol is also called syslog.

This is completely separate from SNMP, which is the usual protocol for network-wide monitoring. A syslog arrangement passes and stores text messages, which may have any format the sending application likes, whilst SNMP is more structured. There have been a couple of attempts by vendor consortiums to replace SNMP, but the proposed successors haven't really worked out, although Windows uses a varient of one of them, under the name WMI (Windows Management Instrumentation).