I can not remember what it is, or how to use it. I remember using a command a while back on some suspicious processes a few years ago to track down the origin of them.

Here lately on my box, I've been getting a lot of hits on apache, but of processes that look like this

Not too far out of the ordinary, but it's getting lots of them (50+) at once. Not sure what is causing it, but I'd like to track down the process.

When this happens, the server load usually goes up in the 50.0+ range.

The command I was looking for is lsof.

Well I'm not finding what I want to find, that of which I'm not sure what I want to find but it's not helping me solve my problem. So lets start with the basics.

What causes a system's load to raise? CPU usage? RAM usage? Anything else? CPU and RAM are easily measured using TOP, but during this process neither one of those are "through the roof" and in most cases are well below 50% on the CPU and hardly any RAM is being used.

You can start here.

http://en.wikipedia.org/wiki/Load_%28computing%29

lsof just shows you what processes are accessing a given file on your fs. This can be useful for many purposes but it has nothing to do with the system load.

Having a heavy system load is not necessarily a bad thing, it depends on how do you work, the kind of software you use and your hardware.

Note that apache can use threads, note as well that your web page probably has some php code on it, which probably adds some more processes for the php interpreter, probably mysql and who knows what else. Web servers nowadays can create a very significant load since they spawn lots of related processes for one or another task.

Then how would you suggest I track down the reason why my server load sometimes gets really high?

High load means usually that tasks are getting stuck, due to a bottleneck of some kind, or due to a program that forks or do more threading than it should. So, it's either that your software is behaving in a buggy way, OR you miscalculated the hardware power that you need to run your work load. There could be a misconfiguration problem as well.

When you have really high loads you should see quite a lot of processes stuck. I'd try turning off all the services, including your web server, mysql, and anything that's not mandatory (very few services are really mandatory). Once you have done that, open top, htop or a similar tool, and work as usual, be vigilant and see if something strange happens. After that, go adding services. I suggested apache as a possible offender because it likes to spawn lots of threads for any random purpose, and bad programming practices and buggy web pages can render your server useless if they produce a lot of threading. An exploitable bug in apache could have a similar effect as well. I assume your machine is rootkit-free and only you have login access to it, else that's the first you should check.

I'd like to believe that my server hasn't been rootkitted, as I am the only one with root level access to it.

As for shutting down non required services such as the ones you mentioned, lets say I do end up releasing the load instantly after I shut down say apache, so now I know that apache is the culprit. What can I do to find out what is causing apache to run the server into over time?

I have a similar problem. Box goes into near-lockup, yet top shows no excessive anything. All I get is insane lag and constant hdd activity. I have yet to figure this out.

You can swap the document-root for your web server to point to anotherlocation with just a simple index.html file on it (no php, not anything else, just plain and simple html or text). That will show you if the problem is specific to your apache installation (or php or whatever else) or it depends on the contents of your web.

I have no idea how complex your web is, and I have no idea of the volume of visits that you have usually. So I can't know if that load is normal (and you are in fact operating over the capabilities of your hardware) or if there's a real problem, a bug in apache or your web site or it's an attack of some kind.

You should start analyzing the output of ps -lA, note down the PID for apache, then start checking the rest of the processes and look for those that have the apache PID on the PPID (parent PID) field. Those will be the apache children. Usually you will see things like php-cgi, mysql and the like. Note that some of these might descend from a previous incarnation of php-cgi and not directly from apache.

Once you've found all the children you can start figuring out what the problem is. For example, if you have 500 instances of php-cgi then the problem is related to php. That's only informative, the problem could still be in your configuration, in your web, in php or even in apache.

If your web has many parts, like a forum, an image gallery, a blog, etc. etc., you should as well try to disable some of these and see what happens. Also, if you have some kind of automated posting or registration mechanism you should make sure you have sane timings configured so spambots don't try to register or login every 2 seconds.

I run Web Host Manager/cPanel on my box with around 14 web sites. Total I probably receive around 3 million to 3.5 million page hits a month. Server is an Intel e6750, 2Gb ram, 500Gb hard drive, 250Gb hard drive and a 100Mbps port speed. I've also noticed the high loads during off peak times of the day (2am for example) when not many people are usually viewing the site. Peak hours are usually around 6 - 10PM CST, since one of my biggest sites (~1 million page views a month) is a local car site, the other busiest site is also a local car forum that's ~2 hour time difference that also receives around 1 million page views a month.

I will do what you suggest though, next time I am on while the server's load is really high.

I am no specialist in web performance by any means. There's a chance that the requirements of the sites you are running are over the top of your hardware, after all you are running 14 sites on a single box, and there's the chance that lots of persons are accessing it at a given time. Sites with dynamic stuff can be very heavy in cpu and ram requirements. For example, imagine the user is uploading or downloading an image or viewing a gallery, php-cgi calls imagemagick or gd (or whatever else) to do the transformations. These are quite heavy in cpu and ram terms. Imagine that 10 users are viewing a gallery at the same time, your cpu can get really busy.

The real deals here are:

• is all the load legitimate or is there an unwanted app (rootkit or whatever else) wasting your cpu cycles sending spam or the like?
• is there some site in the middle that is causing heavy load due to a bug?

Identifying the processes as said above could bring a bit of light into the matter. If the offender is apache or php, identifying the responsible web site which is causing the problem will also be very useful.

You might want to start running the sysstat (sar, iostat, etc.) utilities to see if you can track down the problem, that's probably your best option. They're most likely already installed on your server.

Well I think it's apache. The server's load just started to climb, it got to around 8.0 when I shut down apache completely. As soon as I did that, the server load started to come back down.

Now I just need to figure out why. Now I need to see which actual site is the culprit, and/or which script is causing the problems. How do I go about in doing that?

I'd first check post #9 again, try to change your document-root to point to an empty index.html file and see if apache still goes mad. In that case it's probably some bug in your apache installation. If not you are probably going to need to move all the sites elsewhere, and then put them back one by one and see what happens.

Well I found something interesting when I ran the netstat command.

Lot of connections from 66.215.210.46. I used iptables to block all incoming connections from that IP and so far the server load hasn't spiked. Is that normal to have that many simultaneous connections from the same host?