Has anyone run into the following?

Linux (Fedora Core 6)
Apache 2.2.5
MySQL 5.X
PHP 5.1.6

This is a VPS that will occasionally become inaccessible.

I am hosting multiple websites via Virtualmin with the usual assortment of servers (dovecot, iptables, SSH ). Every few days the machine will stop serving pages and no longer respond to SSH. I can still ping it and get slower response (100+ms).

I email my hosting company to have it restart and dig through the logs but can't find anything noteworthy. In checking out the httpd log there was no mention that the machine crashed or anything just an entry when the web server started again.

Any ideas what it is or where to look?

Jim

Can you do an nmap of it to see if anything is still listening? I've had servers that do this and it can be extremely irritating to debug. One thing I've been experimenting with lately is setting up netdump to log kernel messages/panics/oopses to a remote system for later analysis. I don't know if this is possible with a virtualized server, but suspect it might be.

The problem could also be with the host machine. Can you ask your hosting company if other servers on the box have had similar issues?

It might also be the server is collapsing under load. Does your provided give you bandwidth graphs? If not you might want to set up something like MRTG or bandwidth to see what the bandwidth profile looks like right before the lock-up.

Thanks for the suggestions btmiller.
I like the idea of remotely storing log files. I'll also contact my host to see if anyone else is reporting hardware issues.

Last night my machine seemed to go unresponsive for a few hours before comming back up. In looking through my logs I ran across this in my message file right before it went down.

Does this mean the server ran out of memory? Was it apache that caused the problem? Apache log is below.

messages

Feb 19 01:00:59 vps245 kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Feb 19 01:00:59 vps245 kernel:
Feb 19 01:00:59 vps245 kernel: Call Trace:
Feb 19 01:00:59 vps245 kernel: [<ffffffff802b27c1>] out_of_memory+0x6d/0x1ce
Feb 19 01:04:06 vps245 kernel: [<ffffffff8020f4d2>] __alloc_pages+0x238/0x2c1
Feb 19 01:04:14 vps245 kernel: [<ffffffff80212fd9>] __do_page_cache_readahead+0xd4/0x23a
Feb 19 01:04:14 vps245 kernel: [<ffffffff802603a9>] __wait_on_bit_lock+0x5b/0x66
Feb 19 01:04:14 vps245 kernel: [<ffffffff88061c52>] :dm_mod:dm_any_congested+0x3b/0x42
Feb 19 01:04:14 vps245 kernel: [<ffffffff802138f2>] filemap_nopage+0x148/0x34e
Feb 19 01:04:14 vps245 kernel: [<ffffffff80208ce2>] __handle_mm_fault+0x3ef/0xf9e
Feb 19 01:04:14 vps245 kernel: [<ffffffff802616c8>] _spin_lock_irqsave+0x1a/0x23
Feb 19 01:04:14 vps245 kernel: [<ffffffff80264102>] do_page_fault+0xe59/0x11f8
Feb 19 01:04:14 vps245 kernel: [<ffffffff8025fa14>] thread_return+0x64/0xfe
Feb 19 01:05:16 vps245 kernel: [<ffffffff802616c8>] _spin_lock_irqsave+0x1a/0x23
Feb 19 01:06:45 vps245 kernel: [<ffffffff80261d57>] error_exit+0x0/0x6e
Feb 19 01:07:02 vps245 kernel:
Feb 19 01:07:02 vps245 kernel: Mem-info:
Feb 19 01:07:02 vps245 kernel: DMA per-cpu:
Feb 19 01:07:02 vps245 kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 13 Cold: hi: 62, btch: 15 usd: 23
Feb 19 01:07:02 vps245 kernel: Active:86792 inactive:79793 dirty:0 writeback:0 unstable:0 free:1304 slab:4141 mapped:48 pagetables:13088
Feb 19 01:07:02 vps245 kernel: DMA free:5216kB min:3524kB low:4404kB high:5284kB active:347168kB inactive:319172kB present:777556kB pages_scanned:1221750 all_unreclaimable? yes
Feb 19 01:07:02 vps245 kernel: lowmem_reserve[]: 0 0 0
Feb 19 01:07:06 vps245 kernel: DMA: 422*4kB 11*8kB 3*16kB 2*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 5216kB
Feb 19 01:10:05 vps245 kernel: Swap cache: add 622757, delete 622757, find 55676/100938, race 0+27
Feb 19 01:10:06 vps245 kernel: Free swap = 0kB
Feb 19 01:10:43 vps245 kernel: Total swap = 819192kB
Feb 19 01:10:51 vps245 kernel: Free swap: 0kB
Feb 19 01:10:59 vps245 kernel: 198656 pages of RAM
Feb 19 01:11:15 vps245 kernel: 5092 reserved pages
Feb 19 01:12:05 vps245 kernel: 24236 pages shared
Feb 19 01:12:14 vps245 kernel: 0 pages swap cached
Feb 19 01:12:19 vps245 kernel: Out of Memory: Kill process 1473 (mysqld_safe) score 318273 and children.
Feb 19 01:12:20 vps245 kernel: Out of memory: Killed process 1506 (mysqld).

Apache
[Sun Feb 18 23:03:15 2007] [notice] Graceful restart requested, doing restart
[Sun Feb 18 23:03:15 2007] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)
[Sun Feb 18 23:03:16 2007] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Feb 18 23:03:16 2007] [notice] Apache/2.2.3 (Fedora) configured -- resuming normal operations
[Sun Feb 18 23:03:26 2007] [notice] caught SIGTERM, shutting down
[Sun Feb 18 23:03:28 2007] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Feb 18 23:03:28 2007] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Feb 18 23:03:28 2007] [notice] Apache/2.2.3 (Fedora) configured -- resuming normal operations
[Mon Feb 19 02:25:44 2007] [error] mod_fcgid: fcgid process manager died, restarting the server
[Mon Feb 19 02:25:52 2007] [warn] child process 3802 still did not exit, sending a SIGTERM
[Mon Feb 19 02:25:57 2007] [warn] child process 3803 still did not exit, sending a SIGTERM
[Mon Feb 19 02:25:57 2007] [warn] child process 3804 still did not exit, sending a SIGTERM

From the kernel log it does look like the system ran out of memory and the kernel had to kill mysqld. It looks like either your virtual machine does not have enough memory to handle the load on it or someone is exploiting one of your Web scripts to exhaust the memory on the machine. Without knowing more about the application pfofile, it's hard to say exactly what;s going on.

Why don't you look at top and free to see how much memory your system uses and try to extrapolate if it could become overloaded.

What puzzles me is that I purchased a higher end machine just in case it was a memory shortage and I'm still having similar problems.

I have the commercial version of Virtualmin installed and am running 3 drupal sites, 1 modx site, 1 smf forum and 1 homegrown applications. So it could be a hacker But how would I know what script is causing memory problems? I wonder if I can limit the damage via apache or php directives?

Great idea, I'm just not sure how to extrapolate if it could become overloaded. I have a hard enough time reading "top". From what I understand it can overestimate memory usage. Still I should do some reading since that's part of the fun in having my own linux server right!

BTW FREE shows this a couple hours after rebooting
.................total......used.......free..........shared...buffers...cached
Mem:.........786432.....642068.....144364........0......19152.....225120
-/+ buffers/cache:......397796.....388636
Swap:........819192......0..........819192