Out of memory problem

proNick · 11-22-2013, 04:51 AM

Hello,

From time to time our server freezes.

Quick tail check of /var/log/messages returned next output:

Code:

Nov 22 05:01:42 um-1027 kernel: Node 0 DMA32 free:3988kB min:4008kB low:5008kB high:6012kB active:956080kB inactive:4816kB present:1017072kB pages_scanned:1895625 all_unreclaimable? yes
Nov 22 05:01:42 um-1027 kernel: lowmem_reserve[]: 0 0 0 0
Nov 22 05:01:42 um-1027 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Nov 22 05:01:42 um-1027 kernel: lowmem_reserve[]: 0 0 0 0
Nov 22 05:01:42 um-1027 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Nov 22 05:01:42 um-1027 kernel: lowmem_reserve[]: 0 0 0 0
Nov 22 05:01:42 um-1027 kernel: Node 0 DMA: 4*4kB 4*8kB 4*16kB 3*32kB 2*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2256kB
Nov 22 05:01:42 um-1027 kernel: Node 0 DMA32: 15*4kB 5*8kB 3*16kB 0*32kB 14*64kB 1*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3988kB
Nov 22 05:01:42 um-1027 kernel: Node 0 Normal: empty
Nov 22 05:01:42 um-1027 kernel: Node 0 HighMem: empty
Nov 22 05:01:42 um-1027 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov 22 05:01:42 um-1027 kernel: Free swap  = 0kB
Nov 22 05:01:42 um-1027 kernel: Total swap = 0kB
Nov 22 05:01:42 um-1027 kernel: Free swap:            0kB
Nov 22 05:01:42 um-1027 kernel: 261888 pages of RAM
Nov 22 05:01:42 um-1027 kernel: 5353 reserved pages
Nov 22 05:01:42 um-1027 kernel: 44677 pages shared
Nov 22 05:01:42 um-1027 kernel: 0 pages swap cached
Nov 22 05:01:42 um-1027 kernel: Out of memory: Killed process 1370 (clamd).
Nov 22 05:09:07 um-1027 kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Nov 22 05:09:07 um-1027 kernel: 
Nov 22 05:09:07 um-1027 kernel: Call Trace:
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff800bffdf>] out_of_memory+0x8e/0x2f5
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff8000f082>] __alloc_pages+0x22b/0x2b4
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff80012741>] __do_page_cache_readahead+0x95/0x1d9
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff800628e1>] __wait_on_bit_lock+0x5b/0x66
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff8003f13e>] __lock_page+0x5e/0x64
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff800130cc>] filemap_nopage+0x148/0x322
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff800087ed>] __handle_mm_fault+0x1f8/0xdf4
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff80065a6a>] do_page_fault+0x4b8/0x81d
Nov 22 05:09:07 um-1027 kernel:  [<ffffffff8005cde9>] error_exit+0x0/0x84
Nov 22 05:09:07 um-1027 kernel: 
Nov 22 05:09:09 um-1027 kernel: Mem-info:
Nov 22 05:09:10 um-1027 kernel: Node 0 DMA per-cpu:
Nov 22 05:09:10 um-1027 kernel: cpu 0 hot: high 0, batch 1 used:0
Nov 22 05:09:10 um-1027 kernel: cpu 0 cold: high 0, batch 1 used:0
Nov 22 05:09:10 um-1027 kernel: cpu 1 hot: high 0, batch 1 used:0
Nov 22 05:09:10 um-1027 kernel: cpu 1 cold: high 0, batch 1 used:0
Nov 22 05:09:10 um-1027 kernel: Node 0 DMA32 per-cpu:
Nov 22 05:09:10 um-1027 kernel: cpu 0 hot: high 186, batch 31 used:162
Nov 22 05:09:10 um-1027 kernel: cpu 0 cold: high 62, batch 15 used:50
Nov 22 05:09:10 um-1027 kernel: cpu 1 hot: high 186, batch 31 used:57
Nov 22 05:09:10 um-1027 kernel: cpu 1 cold: high 62, batch 15 used:17
Nov 22 05:09:10 um-1027 kernel: Node 0 Normal per-cpu: empty
Nov 22 05:09:10 um-1027 kernel: Node 0 HighMem per-cpu: empty
Nov 22 05:09:10 um-1027 kernel: Free pages:        6248kB (0kB HighMem)
Nov 22 05:09:10 um-1027 kernel: Active:239484 inactive:347 dirty:0 writeback:0 unstable:0 free:1562 slab:4426 mapped-file:973 mapped-anon:239090 pagetables:5625
Nov 22 05:09:10 um-1027 kernel: Node 0 DMA free:2256kB min:40kB low:48kB high:60kB active:0kB inactive:0kB present:10824kB pages_scanned:0 all_unreclaimable? yes
Nov 22 05:09:10 um-1027 kernel: lowmem_reserve[]: 0 993 993 993
Nov 22 05:09:10 um-1027 kernel: Node 0 DMA32 free:3992kB min:4008kB low:5008kB high:6012kB active:958048kB inactive:1260kB present:1017072kB pages_scanned:2140318 all_unreclaimable? yes
Nov 22 05:09:10 um-1027 kernel: lowmem_reserve[]: 0 0 0 0
Nov 22 05:09:10 um-1027 kernel: Node 0 Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Nov 22 05:09:10 um-1027 kernel: lowmem_reserve[]: 0 0 0 0
Nov 22 05:09:10 um-1027 kernel: Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
Nov 22 05:09:10 um-1027 kernel: lowmem_reserve[]: 0 0 0 0
Nov 22 05:09:10 um-1027 kernel: Node 0 DMA: 4*4kB 4*8kB 4*16kB 3*32kB 2*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2256kB
Nov 22 05:09:11 um-1027 kernel: Node 0 DMA32: 14*4kB 8*8kB 4*16kB 1*32kB 3*64kB 6*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3992kB
Nov 22 05:09:11 um-1027 kernel: Node 0 Normal: empty
Nov 22 05:09:11 um-1027 kernel: Node 0 HighMem: empty
Nov 22 05:09:11 um-1027 kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov 22 05:09:11 um-1027 kernel: Free swap  = 0kB
Nov 22 05:09:11 um-1027 kernel: Total swap = 0kB
Nov 22 05:09:11 um-1027 kernel: Free swap:            0kB
Nov 22 05:09:11 um-1027 kernel: 261888 pages of RAM
Nov 22 05:09:11 um-1027 kernel: 5353 reserved pages
Nov 22 05:09:11 um-1027 kernel: 44992 pages shared
Nov 22 05:09:11 um-1027 kernel: 0 pages swap cached
Nov 22 05:09:11 um-1027 kernel: Out of memory: Killed process 1893 (clamd).

Looks like there is a problem with calmd, but I suspect that it may be hardware problem with memory... Can you give me some suggestions what to check more to locate the problem?

syg00 · 11-22-2013, 05:30 AM

OOM-killer more likely means software, not hardware, problem.
Looks like a 32-bit system with not much memory, and no swap.

So:
1) define some swap space - start with 4 Gig.
2) add some RAM.
3) get a 64-bit system with a decent amount of RAM - these days you probably can't buy less than 8 Gig.

johnsfine · 11-22-2013, 07:17 AM

Quote:

Originally Posted by syg00

Looks like a 32-bit system with not much memory, and no swap.

Where did you get the idea it was a 32-bit system? There was a 64-bit stack trace, which I think is unlikely in a 32-bit system.

I agree with "not much memory and no swap".

Quote:

1) define some swap space - start with 4 Gig.

Yes.

Quote:

Originally Posted by proNick

Can you give me some suggestions what to check more to locate the problem?

Is this a physical machine or some kind of VM? The low amount of ram and lack of swap seems unlikely for a physical machine.

If it is a VM, it is probably under configured for the task you are trying to do (though probably the task could be tuned to use less memory).

If it is a physical machine, does it really have so little RAM? Or do you have some BIOS setting wrong stopping the OS from seeing the rest of the ram? In any case, if it is a physical machine you should configure some swap space. If it is a VM, maybe you should configure some swap space. That depends on what kind of VM.

proNick · 11-22-2013, 07:18 AM

what confuses me is that this is few years old server, and there was no issues like this in the past.

johnsfine · 11-22-2013, 07:28 AM

Quote:

Originally Posted by proNick

what confuses me is that this is few years old server, and there was no issues like this in the past.

Do you know how much ram and how much swap space it had when it was working? In other words, has some hardware or software malfunction reduced the available resources, or has something increased the used resources?

syg00 · 11-22-2013, 03:24 PM

Quote:

Originally Posted by johnsfine

Where did you get the idea it was a 32-bit system?

My bad - I thought DMA32 went away on 64-bit, and just didn't look properly.

As to "why now"?: software update, more traffic, long(er) uptime ... - lots of possibilities.

proNick · 11-27-2013, 05:09 AM

hello,

this one is still active...

@syg00 - how did you know that i don't have swap partition?

and also, i'm bit confused, because this server worked few years without problems like this.

i'm able to free up some memory (or clear cache) by running next command:

Code:

sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"

... but this is not real solution.

proNick · 11-27-2013, 05:44 AM

also, after running command

ps aux

... result i'm getting is:

Code:

[root@um-1027 ~]# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  10312   164 ?        Ss   Nov26   0:00 init [3]                   
root         2  0.0  0.0      0     0 ?        S    Nov26   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   Nov26   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    Nov26   0:00 [watchdog/0]
root         5  0.0  0.0      0     0 ?        S    Nov26   0:00 [migration/1]
root         6  0.0  0.0      0     0 ?        SN   Nov26   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S    Nov26   0:00 [watchdog/1]
root         8  0.0  0.0      0     0 ?        S<   Nov26   0:00 [events/0]
root         9  0.0  0.0      0     0 ?        S<   Nov26   0:00 [events/1]
root        10  0.0  0.0      0     0 ?        S<   Nov26   0:00 [khelper]
root        19  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kthread]
root        24  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kblockd/0]
root        25  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kblockd/1]
root        26  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kacpid]
root       150  0.0  0.0      0     0 ?        S<   Nov26   0:00 [cqueue/0]
root       151  0.0  0.0      0     0 ?        S<   Nov26   0:00 [cqueue/1]
root       154  0.0  0.0      0     0 ?        S<   Nov26   0:00 [khubd]
root       156  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kseriod]
root       229  0.0  0.0      0     0 ?        S    Nov26   0:01 [pdflush]
root       230  0.0  0.0      0     0 ?        S    Nov26   0:00 [pdflush]
root       231  0.0  0.0      0     0 ?        S<   Nov26   0:09 [kswapd0]
root       232  0.0  0.0      0     0 ?        S<   Nov26   0:00 [aio/0]
root       233  0.0  0.0      0     0 ?        S<   Nov26   0:00 [aio/1]
root       386  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kpsmoused]
root       420  0.0  0.0      0     0 ?        S<   Nov26   0:00 [ata/0]
root       421  0.0  0.0      0     0 ?        S<   Nov26   0:00 [ata/1]
root       422  0.0  0.0      0     0 ?        S<   Nov26   0:00 [ata_aux]
root       426  0.0  0.0      0     0 ?        S<   Nov26   0:00 [scsi_eh_0]
root       427  0.0  0.0      0     0 ?        S<   Nov26   0:00 [scsi_eh_1]
root       433  0.0  0.0      0     0 ?        S<   Nov26   0:09 [md1_raid1]
root       436  0.0  0.0      0     0 ?        S<   Nov26   0:00 [md0_raid1]
root       437  0.0  0.0      0     0 ?        S<   Nov26   0:06 [kjournald]
root       464  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kauditd]
root       498  0.0  0.0  12616   432 ?        S<s  Nov26   0:00 /sbin/udevd -d
vpopmail   762  0.0  0.0  10124   768 ?        S    02:13   0:02 /usr/bin/imapd Maildir
root      1595  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kmpathd/0]
root      1596  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kmpathd/1]
root      1621  0.0  0.0      0     0 ?        S<   Nov26   0:00 [kjournald]
root      2138  0.0  0.0  16240   368 ?        S<sl Nov26   0:00 auditd
root      2140  0.0  0.2 123776  2868 ?        S<s  Nov26   0:00 python /sbin/audispd
root      2170  0.0  0.0   5872   244 ?        Ss   Nov26   0:00 syslogd -m 0
root      2173  0.0  0.0   3772   100 ?        Ss   Nov26   0:00 klogd -x
rpc       2224  0.0  0.0   8020   136 ?        Ss   Nov26   0:00 portmap
root      2249  0.0  0.0   8020   144 ?        Ss   Nov26   0:00 rpc.statd
root      2298  0.0  0.0   3952   240 ?        Ss   Nov26   0:00 mdadm --monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid
root      2332  0.0  0.0  48616   464 ?        Ss   Nov26   0:00 rpc.idmapd
dbus      2359  0.0  0.0  21228   260 ?        Ss   Nov26   0:00 dbus-daemon --system
root      2406  0.0  0.0  20996   956 ?        Ssl  Nov26   0:00 pcscd
root      2440  0.0  0.0   8480   120 ?        Ss   Nov26   0:00 /usr/bin/hidd --server
root      2460  0.0  0.0  29932   588 ?        Ssl  Nov26   0:00 automount
root      2483  0.0  0.0   3772   124 ?        Ss   Nov26   0:00 /usr/sbin/acpid
root      2511  0.0  0.0   3796   140 ?        S    Nov26   0:00 svscan
root      2527  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise smtp
root      2528  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise log
root      2529  0.0  0.0   3624    84 ?        S    Nov26   0:00 supervise imap4
root      2530  0.0  0.0   3624    84 ?        S    Nov26   0:00 supervise log
root      2531  0.0  0.0   3624    84 ?        S    Nov26   0:00 supervise pop3
root      2532  0.0  0.0   3624    84 ?        S    Nov26   0:00 supervise log
root      2533  0.0  0.0   3624    84 ?        S    Nov26   0:00 supervise submission
root      2535  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise log
root      2536  0.0  0.0   3624    88 ?        S    Nov26   0:00 supervise pop3-ssl
root      2537  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise log
root      2538  0.0  0.0   3624    84 ?        S    Nov26   0:00 supervise authlib
root      2539  0.0  0.0   3624    84 ?        S    Nov26   0:00 supervise log
root      2540  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise clamd
root      2541  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise log
root      2542  0.0  0.0   3624    84 ?        S    Nov26   0:00 supervise imap4-ssl
root      2543  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise log
root      2544  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise send
root      2545  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise log
root      2546  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise spamd
root      2547  0.0  0.0   3620    80 ?        S    Nov26   0:00 supervise log
vpopmail  2549  0.0  0.0   6776   148 ?        S    Nov26   0:00 /usr/bin/tcpserver -v -R -H -l um-1027.conepuppy.com -x /etc/tcprules.d/tcp.smtp.cdb -c 300 -u 89 -g 89 0 smtp /usr/bin/rblsmtpd -r bl.spamcop.net -r zen.spamhaus.org /var/qmail/bin/qmail-smtpd /home/vpopm
qmaill    2550  0.0  0.0   3768   112 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/submission
root      2551  0.0  0.0  52376   476 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2552  0.0  0.0   4684   136 ?        S    Nov26   0:01 /usr/bin/tcpserver -v -c 200 -R -H -l um-1027.conepuppy.com 0 110 /usr/sbin/pop3login /usr/bin/pop3d Maildir
qmaill    2553  0.0  0.0   3636    64 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/authlib
qmaill    2554  0.0  0.0   3772   116 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/smtp
qmaill    2561  0.0  0.0   3768   116 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/imap4
root      2568  0.0  0.0  57588   724 ?        Ss   Nov26   0:00 /usr/sbin/sshd
vpopmail  2577  0.0  0.0   4684   136 ?        S    Nov26   0:00 /usr/bin/tcpserver -v -R -H -l um-1027.conepuppy.com -x /etc/tcprules.d/tcp.smtp.cdb -c 300 -u 89 -g 89 0 587 /var/qmail/bin/qmail-smtpd /home/vpopmail/bin/vchkpw /bin/true
qmaill    2578  0.0  0.0   3768   164 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/pop3
qmaill    2579  0.0  0.0   3636    96 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/imap4-ssl
root      2581  0.0  0.0   4684   136 ?        S    Nov26   0:00 /usr/bin/tcpserver -v -R -H -l um-1027.conepuppy.com 0 143 /usr/sbin/imaplogin /usr/bin/imapd Maildir
qmaill    2582  0.0  0.0   3636    96 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/pop3-ssl
root      2583  0.0  0.0   4684   132 ?        S    Nov26   0:00 /usr/bin/tcpserver -v -R -H -l um-1027.conepuppy.com 0 993 /usr/bin/couriertls -server -tcpd /usr/sbin/imaplogin /usr/bin/imapd Maildir
qmails    2584  0.0  0.0  10160   252 ?        S    Nov26   0:01 qmail-send
qmaill    2585  0.0  0.0   3772   112 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/send
qmaill    2586  0.0  0.0   3640    96 ?        S    Nov26   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/clamd
clamav    2593  0.3 47.5 622292 488232 ?       Sl   Nov26   3:31 /usr/sbin/clamd
root      2594  0.0  0.0   4684   132 ?        S    Nov26   0:00 /usr/bin/tcpserver -v -R -H -l um-1027.conepuppy.com 0 995 /usr/bin/couriertls -server -tcpd /usr/sbin/pop3login /usr/bin/pop3d Maildir
root      2658  0.0  0.0   3784   152 ?        S    Nov26   0:00 qmail-lspawn ./Maildir/
qmailr    2659  0.0  0.0   3780   152 ?        S    Nov26   0:00 qmail-rspawn
qmailq    2660  0.0  0.0   3756   144 ?        S    Nov26   0:00 qmail-clean
root      2667  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2668  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2669  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2670  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2671  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2672  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2673  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2674  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2675  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2676  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2677  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2678  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2679  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
root      2680  0.0  0.0  54588   844 ?        S    Nov26   0:01 /usr/libexec/courier-authlib/authdaemond
root      2681  0.0  0.0  54588   844 ?        S    Nov26   0:00 /usr/libexec/courier-authlib/authdaemond
ntp       2687  0.0  0.4  23320  4952 ?        SLs  Nov26   0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root      2707  0.0  0.0  44152   548 ?        Ss   Nov26   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
clamav    2722  0.0  0.1  39436  1028 ?        Ss   Nov26   0:00 /usr/bin/freshclam -d -p /var/run/clamav/freshclam.pid
root      2789  0.0  0.0  65920   764 ?        S    Nov26   0:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --log-error=/var/log/mysqld.log
mysql     2825  0.1  2.6 263512 27068 ?        Sl   Nov26   2:00 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --socket=/var/lib/mysql/mysql.sock
root      2869  0.0  0.0   6416   320 ?        Ss   Nov26   0:00 gpm -m /dev/input/mice -t exps2
root      2888  0.0  0.4 233316  4960 ?        Ss   Nov26   0:00 /usr/sbin/httpd
root      2903  0.0  0.1  74800  1060 ?        Ss   Nov26   0:00 crond
root      2932  0.0  0.0  18684   444 ?        Ss   Nov26   0:00 /usr/sbin/atd
apache    2947  0.0  1.5 297700 16000 ?        S    Nov26   0:22 /usr/sbin/httpd
apache    2952  0.0  1.5 299808 16188 ?        S    Nov26   0:22 /usr/sbin/httpd
root      2957  0.0  4.6 272876 47660 ?        S    Nov26   0:59 /usr/bin/python /usr/sbin/yum-updatesd
avahi     2973  0.0  0.0  24116   808 ?        Ss   Nov26   0:00 avahi-daemon: running [um-1027.local]
avahi     2974  0.0  0.0  23092   264 ?        Ss   Nov26   0:00 avahi-daemon: chroot helper
68        2989  0.0  0.2  30552  2840 ?        Ss   Nov26   0:00 hald
root      2990  0.0  0.0  21632   636 ?        S    Nov26   0:00 hald-runner
68        2997  0.0  0.0  12272   624 ?        S    Nov26   0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
68        3006  0.0  0.0  12272   632 ?        S    Nov26   0:00 hald-addon-keyboard: listening on /dev/input/event1
68        3010  0.0  0.0  12276   632 ?        S    Nov26   0:00 hald-addon-keyboard: listening on /dev/input/event0
root      3056  0.0  0.0  18384   476 ?        S    Nov26   0:00 /usr/sbin/smartd -q never
root      3059  0.0  0.0   3760   436 tty1     Ss+  Nov26   0:00 /sbin/mingetty tty1
root      3060  0.0  0.0   3760   436 tty2     Ss+  Nov26   0:00 /sbin/mingetty tty2
root      3061  0.0  0.0   3760   436 tty3     Ss+  Nov26   0:00 /sbin/mingetty tty3
root      3064  0.0  0.0   3756   432 tty4     Ss+  Nov26   0:00 /sbin/mingetty tty4
root      3067  0.0  0.0   3760   436 tty5     Ss+  Nov26   0:00 /sbin/mingetty tty5
root      3070  0.0  0.0   3756   432 tty6     Ss+  Nov26   0:00 /sbin/mingetty tty6
apache    6088  0.0  1.5 297712 15844 ?        S    Nov26   0:21 /usr/sbin/httpd
apache    6092  0.0  1.5 297640 15864 ?        S    Nov26   0:22 /usr/sbin/httpd
apache    6118  0.0  1.5 297404 15616 ?        S    Nov26   0:21 /usr/sbin/httpd
apache    6285  0.0  1.5 299560 15868 ?        S    Nov26   0:21 /usr/sbin/httpd
apache    6287  0.0  1.6 300764 16960 ?        S    Nov26   0:23 /usr/sbin/httpd
apache    6289  0.0  1.5 297536 15704 ?        S    Nov26   0:23 /usr/sbin/httpd
apache    6290  0.0  1.5 297888 15952 ?        S    Nov26   0:19 /usr/sbin/httpd
apache    6423  0.0  1.4 297204 15296 ?        S    Nov26   0:21 /usr/sbin/httpd
apache    7554  0.0  1.5 297248 15584 ?        S    03:52   0:04 /usr/sbin/httpd
vpopmail  8058  0.0  0.0  10128   776 ?        S    Nov26   0:00 /usr/bin/imapd Maildir
qmaill    8629  0.0  0.0   3636   224 ?        S    04:03   0:00 /usr/bin/multilog t s1000000 n100 /var/log/qmail/spamd
root      8634  0.0  5.1 139544 53024 ?        S    04:03   0:01 /usr/bin/perl -T -w /usr/bin/spamd -x -u vpopmail -s stderr
vpopmail  8646  0.9  6.7 155164 69456 ?        S    04:03   1:30 spamd child
vpopmail  8647  0.0  6.1 148740 62936 ?        S    04:03   0:05 spamd child
root     16176  0.0  0.2  83052  2496 ?        Ss   06:02   0:00 sshd: root@pts/0 
root     16205  0.0  0.1  66056  1428 pts/0    Ss   06:03   0:00 -bash
vpopmail 18256  0.0  0.0  10124   996 ?        S    06:31   0:00 /usr/bin/imapd Maildir
vpopmail 18282  0.0  0.0  10124   596 ?        S    06:31   0:00 /usr/bin/imapd Maildir
root     19355  0.0  0.2  58932  2596 ?        Ss   06:41   0:00 sshd: [accepted] 
sshd     19356  0.0  0.1  58932  1292 ?        S    06:41   0:00 sshd: [net]      
root     19357  0.0  0.0  65592   964 pts/0    R+   06:41   0:00 ps aux
apache   21398  0.0  1.5 297472 15748 ?        S    Nov26   0:13 /usr/sbin/httpd
vpopmail 22054  0.0  0.0   9232   792 ?        S    Nov26   0:00 /usr/bin/imapd Maildir
apache   24465  0.0  1.5 297880 16052 ?        S    00:01   0:13 /usr/sbin/httpd
vpopmail 25030  0.0  0.0   9236   848 ?        S    00:10   0:00 /usr/bin/imapd Maildir
apache   25562  0.0  1.5 297252 15572 ?        S    00:21   0:10 /usr/sbin/httpd

so, it looks that the biggest problem is:

Code:

clamav    2593  0.3 47.5 622292 488232 ?       Sl   Nov26   3:31 /usr/sbin/clamd

Please for some advices...