All I know about Linux I have learned along the way, so in trouble-shooting it is very easy for me to overlook something obvious. Please let me know if I am doing that in this case.

Here is the problem--

I have websites hosted on a Linux VPS server. The VPS has 512 MB RAM. I have only a few modest websites on the server. I have uploaded graphs for a typical daily server load below:

(memory load)
http://www.wildjot.com/server/sonar_mem.png

(process load)
http://www.wildjot.com/server/sonar_load.png

The red point on the graph is most recent instance when the server went down. This server failure has been happening intermittently, on average about once a week though it is not consistently on one day of the week, or even one particular time of the day. I have not (at present) been able to associate the server failure with any particular traffic pattern.

The message I get from my host when the server goes down is this:

Apache was reporting down on your VPS in our monitoring system. We found that Virtuozzo, the management software for our VPS product, killed Apache. This occurred because Apache was attempting to use more memory than your instance is allowed which causes significant instability on the parent server. We have restarted Apache.

If you look at the memory graph I linked above the record for memory use immediately before the server went down is nowhere near my 512 MB limit. When I pointed this out to tech support they said the monitoring system only checks every 5 min so if some uncontrolled memory usage occurred within the 5 min span between checks the server would go down before the high usage was logged in the graph.

Further, the tech pointed out that while Apache was killed for attempting to use memory over my allocation Apache may not be the source of the problem. If some other program (for example, clamd) suddenly started using a huge amount of my memory and then Apache asked for a small amount that put the total over my limit, Apache would be blamed, even if it wasn't the program using the majority of memory. So I don't actually know if Apache is the underlying cause or not.

This pattern of the monitoring graph showing moderate memory usage with the server to suddenly failing about once a week has been going on for months. I am feeling completely baffled because the server normally functions very well and I have poured over both the Apache error log and the domain access logs and at this point I don't feel like I can find anything there to account for the problem.

If I go to my Web Host Manager I can pull up a process manager page which shows cpu and memory usage by processes. It seems like this information ought to tell me what is the source of the problem--if I could look at the information at the time of the server failure. Unfortunately, the time of server failure is so random (and often the middle of the night) that I am never sitting at the computer watching the process manager page when the server goes down.

I can tell you that of this writing the primary users of memory according to that process manager page are clamd (16.6% of mem) mysql (8% mem) and spamd with a child (10.3%)

So my question is this: Is there a log which records that process manager cpu and memory usage? Or is there a way to create such a log?

Beyond the Apache error log, and domain access logs, are there any other logs I should be looking at? Anything else I should do?

Thanks,

Hi,

You can try to catpure the output of a "top" command in a file.

For example:
crontab -e
*/1 * * * * top -b -n1 >> <path_to_your_log_file> 2>&1

1 members found this post helpful.

Thanks, that looks like what I want to do.

I have never done crontab before, so this will be a learning experience. To understand your command I went to http://www.adminschoice.com/docs/crontab.htm (Should anyone else down the road need help deciphering the command)

After looking at that page, I have two minor questions:

(1) Do I need to run this crontab as root? (That may be a stupid question)

(2) Do I need to specify a full absolute path for the log file? Or if I ran it on a user call Boy1 and specified a path of /mem-log/logfile would I then find it at /home/Boy1/mem-log/logfile ?

(3) I presume the crontab command will keep adding to the log file? (I would need to delete it periodically in that case)

Thanks for all your help.

Better to use something already available - see if you have SYSSTAT installed or if it can be. Maintains historical data, and newer versions have pidstat. See here for the online doco.

Using 'atop' as a daemon seems easier to me. Its saved binary logs can be easily replayed later on BTW.

Thank you all for the various suggestions. Now I just need to start logging and then hopefully I will figure out what is causing my problem.

How much swap space does it have?

The label there says
Used Physical Memory (with Buffers and Cache)
If it means what it says, that is a very strange thing to measure and a very strange result from measuring it.

Cache is supposed to use most of the physical memory that isn't in use at the moment by anything else. In a system with just 512 MB ram the total fraction of ram in use including cache should be both higher and more stable than that graph.

That sounds like a design flaw in Virtuozzo and/or VPS.

I don't know anything about VPS, but in normal virtualization the virtualized system is more self contained. If it tries to use too much memory, it should swap more or it should fail by itself, but neither should cause any instability in the parent server.

It also is likely that they are measuring the virtual memory use rather than the actual memory use in the automatic decision for Virtuozzo to kill Apache, but reporting some kind of physical memory use the the graph they reported. Virtual memory use is not a meaningful measure on which to base that kind of decision.

I don't think you have any chance of diagnosing what (if anything) is wrong within your portion of the system until you get specific information about what Virtuozzo is measuring to make its decision about what to kill.

I went and looked at my system stats. This is what it said about my memory at present:

-------total--------used------free----shared--buffers--cached
Mem:---524288-----299136--225152-----0-----0---------0
-/+ buffers/cache:---299136---225152
Swap:-----0-------0-------0
Total:--524288---299136---225152

. . . So it appears I don't have a swap. (Which makes me wonder why not?)

My diskspace is below:

Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 50G 6.3G 44G 13% /
simfs 50G 6.3G 44G 13% /tmp
simfs 50G 6.3G 44G 13% /var/tmp

. . . obviously the disk usage is very low. My bandwidth is also a fraction of my allocated amount

. . . If I am understanding the stats I listed above, I don't have a cache . . . right?


They may not have said it very well, but as I understand it my account dies because Virtuozzo keeps it from making the parent unstable. So I think it is pretty self-contained.

You could be right. I don't know anything about virtual memory vs. physical memory, so that is beyond me at present.

Hi,

No, you don't. You may run it under any user.

Yes, You need to specify a full absolute path for the log file.

You have to delete it periodically.

I did a little online reading about VPS after posting. The various online answers to "what is VPS" seem to be intentionally misleading and uninformative. They claim a high degree of stability and independence from other virtual servers on the same host, while describing a basic design that ought to have very little stability or independence.

From your stats, I guess all the swap space and cache space is hidden inside the host.

I still call that the opposite of self contained.

What I read about VPS says it provides two level of memory "guaranteed" and "burst". I expect that inside each virtual server you must see the "burst" level (or the 512MB max itself couldn't be stable). So I expect 512MB is your burst level, not your guaranteed level.

The documentation says that when your memory use is between the guaranteed level and the burst level, Virtuozzo will kill your process based on the workload in other virtual servers on the same host.

Have a look at openvz from which I believe Parallels (the company) and Virtuozzo (the product) grew. I looked at openvz a couple of years back out of interest, but never pursued it, so I can't offer any specific knowledge. They were an early adopter of containers, and to a large degree drove the direction of what eventually became cgroups in the mainline kernel. For virtuozzo VPSs, think chroot jails.
Probably based on Redhat, so SYSSTAT is likely available if not already installed and active.

I did find SYSSTAT already installed and operating on my VPS system. However, it was only logging data every 10 min which, as I stated in an earlier post, is not often enough to catch whatever problem is occurring. I set SYSSTAT to run every 1 min. Hopefully that will catch the problem next time it occurs.

All sar is going to show you is historical trends - even pidstats (if available) isn't going to show you why Apache is such a storage pig. You'll need to see how many processes it has spawned, and any (Apache) logs if available to see what it was actually doing.

What logs could I look at besides the error log for Apache and the domain logs? If I go to my Web Host Manager there is a page for Apache status, which shows what Apache is doing in the moment, but I haven't found a log which contains a record of that info.