I've got a memory problem, and I'm stumped.
The system in question is serving a bunch of low volume non-profit websites, so rebooting is an issue, but can be done late at night.
I have RH 9.0 running 2.4.20-19.9smp, and after a few days of uptime it gets ungodly slow at serving pages due to swapping. The machine has a gig of ram and is not running x or anything like that.
I've looked at migrating to debian, but I'd like to try to get this machine working as the swap is potentially going to cause a lot of issues.
Here is some output from vmstat:
Code:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 1 139464 16224 1580 35096 2 3 1 3 3 1 1 1 4
0 3 4 139464 16304 1600 34904 0 0 234 168 199 392 1 2 97
0 0 1 139464 16308 1536 34996 0 0 61 43 135 244 0 1 99
0 0 1 139464 16120 1684 35144 0 0 575 198 279 660 2 3 95
Now while a page is loading:
Code:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 1 1 139464 13108 1632 34228 2 3 1 3 3 1 1 1 4
0 1 2 139464 10116 1740 35064 886 0 1494 243 290 525 13 3 84
0 3 1 139464 9040 1632 35436 430 0 874 174 280 562 7 4 89
1 0 1 139464 8604 1660 35036 1306 92 1353 217 265 409 13 3 84
0 0 1 139388 8728 1724 35224 338 180 496 244 239 497 15 1 84
So, it's swapping hard. So I look at free:
Code:
total used free shared buffers cached
Mem: 1030304 1020468 9836 0 1444 35336
-/+ buffers/cache: 983688 46616
Swap: 2072376 139388 1932988
That "-/+ buffers/cache:" line looks bad to me, but I'm not sure. I think the buffer line is high for this setup, it's filling most of RAM. Also, there is a lot of swapped data.
So I looked at ps aux:
Code:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1384 440 ? S 2004 1:22 init
root 2 0.0 0.0 0 0 ? SW 2004 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SW 2004 0:00 [migration/1]
root 4 0.0 0.0 0 0 ? SW 2004 13:21 [keventd]
root 5 0.0 0.0 0 0 ? SWN 2004 0:00 [ksoftirqd_CPU0]
root 6 0.0 0.0 0 0 ? SWN 2004 0:00 [ksoftirqd_CPU1]
root 11 0.0 0.0 0 0 ? SW 2004 0:25 [bdflush]
root 7 0.5 0.0 0 0 ? DW 2004 411:25 [kswapd]
root 8 0.0 0.0 0 0 ? SW 2004 0:16 [kscand/DMA]
root 9 0.0 0.0 0 0 ? SW 2004 57:26 [kscand/Normal]
root 10 0.1 0.0 0 0 ? SW 2004 73:05 [kscand/HighMem]
root 12 0.0 0.0 0 0 ? SW 2004 2:02 [kupdated]
root 13 0.0 0.0 0 0 ? SW 2004 0:00 [mdrecoveryd]
root 19 0.0 0.0 0 0 ? SW 2004 0:00 [scsi_eh_0]
root 20 0.0 0.0 0 0 ? SW 2004 0:00 [scsi_eh_1]
root 24 0.0 0.0 0 0 ? SW 2004 6:49 [kjournald]
root 82 0.0 0.0 0 0 ? SW 2004 0:00 [khubd]
root 188 0.0 0.0 0 0 ? SW 2004 0:40 [kjournald]
root 189 0.0 0.0 0 0 ? SW 2004 0:00 [kjournald]
root 190 0.0 0.0 0 0 ? SW 2004 0:29 [kjournald]
root 191 0.0 0.0 0 0 ? SW 2004 12:37 [kjournald]
root 192 0.0 0.0 0 0 ? SW 2004 7:16 [kjournald]
root 692 0.0 0.0 1456 512 ? S 2004 1:01 syslogd -m 0
root 696 0.0 0.0 1376 388 ? S 2004 0:00 klogd -x
rpc 714 0.0 0.0 1560 440 ? S 2004 0:00 portmap
rpcuser 733 0.0 0.0 1580 568 ? S 2004 0:00 rpc.statd -p 4000
root 844 0.0 0.0 3720 484 ? S 2004 1:25 /usr/sbin/sshd
root 860 0.0 0.0 2072 540 ? S 2004 0:00 xinetd -stayalive
ntp 879 0.0 0.2 2412 2408 ? SL 2004 4:27 ntpd -U ntp
lp 895 0.0 0.0 9136 160 ? S 2004 0:19 lpd Waiting
root 906 0.0 0.0 1424 4 ? S 2004 0:00 gpm -t imps2 -m /
bin 927 0.0 0.0 2064 40 ? S 2004 0:01 /usr/sbin/cannase
root 938 0.0 0.0 1432 136 ? S 2004 0:38 crond
xfs 971 0.0 0.0 5664 28 ? S 2004 0:00 xfs -droppriv -da
daemon 1010 0.0 0.0 1420 112 ? S 2004 0:04 /usr/sbin/atd
root 1020 0.0 0.0 3416 388 ? S 2004 0:00 rhnsd --interval
root 1027 0.0 0.0 1360 0 tty2 SW 2004 0:00 /sbin/mingetty tt
root 1028 0.0 0.0 1360 0 tty3 SW 2004 0:00 /sbin/mingetty tt
root 1029 0.0 0.0 1360 0 tty4 SW 2004 0:00 /sbin/mingetty tt
root 1030 0.0 0.0 1360 0 tty5 SW 2004 0:00 /sbin/mingetty tt
root 1031 0.0 0.0 1360 0 tty6 SW 2004 0:00 /sbin/mingetty tt
root 1032 0.0 0.0 2120 0 ? SW 2004 0:00 /bin/sh /command/
root 1034 0.0 0.0 1400 88 ? S 2004 3:48 svscan /service
root 1035 0.0 0.0 1352 0 ? SW 2004 0:00 readproctitle ser
root 1036 0.0 0.0 1352 12 ? S 2004 0:00 supervise qmail-s
root 1037 0.0 0.0 1352 24 ? S 2004 0:00 supervise log
root 1038 0.0 0.0 1352 12 ? S 2004 0:00 supervise qmail-s
root 1039 0.0 0.0 1352 12 ? S 2004 0:00 supervise log
qmaild 1040 0.0 0.0 1428 4 ? S 2004 0:05 /usr/local/bin/tc
qmaill 1041 0.0 0.0 1376 0 ? SW 2004 0:04 /usr/local/bin/mu
qmails 1042 0.0 0.0 1580 216 ? S 2004 63:05 qmail-send
qmaill 1043 0.0 0.0 1376 64 ? S 2004 26:29 /usr/local/bin/mu
root 1046 0.0 0.0 1372 104 ? S 2004 55:27 qmail-lspawn |dot
qmailr 1047 0.0 0.0 1368 0 ? SW 2004 0:05 qmail-rspawn
qmailq 1048 0.0 0.0 1364 52 ? S 2004 1:39 qmail-clean
root 6398 0.0 0.0 1364 4 tty1 S 2004 0:00 /sbin/mingetty tt
root 13626 0.0 0.0 0 0 ? SW 2004 0:17 [kjournald]
root 22547 0.0 0.0 0 0 ? SW Jan02 0:00 [nfsd]
root 22549 0.0 0.0 0 0 ? SW Jan02 0:00 [lockd]
root 22550 0.0 0.0 0 0 ? SW Jan02 0:18 [rpciod]
rpcuser 22552 0.0 0.0 1580 4 ? S Jan02 0:00 /sbin/rpc.statd
root 22554 0.0 0.0 1508 8 ? S Jan02 0:00 /usr/sbin/rpc.mou
root 22556 0.0 0.0 3628 4 ? S Jan02 0:00 /usr/sbin/rpc.rqu
rpcuser 30664 0.0 0.0 1580 28 ? S Jan02 0:00 /sbin/rpc.statd
root 30666 0.0 0.0 1508 0 ? SW Jan02 0:00 /usr/sbin/rpc.mou
root 30668 0.0 0.0 3636 0 ? SW Jan02 0:00 /usr/sbin/rpc.rqu
rpcuser 15749 0.0 0.0 1576 0 ? SW Jan05 0:00 /sbin/rpc.statd
root 15751 0.0 0.0 1508 0 ? SW Jan05 0:00 /usr/sbin/rpc.mou
root 15753 0.0 0.0 3636 0 ? SW Jan05 0:00 /usr/sbin/rpc.rqu
root 19011 0.0 0.0 18652 84 ? S Feb04 0:17 /usr/sbin/httpd -
www-data 27260 0.0 0.4 26720 4760 ? S Feb06 2:21 /usr/sbin/httpd -
www-data 27261 0.0 0.8 26472 8316 ? S Feb06 2:13 /usr/sbin/httpd -
www-data 27267 0.0 0.5 26640 5168 ? S Feb06 2:24 /usr/sbin/httpd -
www-data 27271 0.0 0.5 26724 5640 ? S Feb06 2:23 /usr/sbin/httpd -
www-data 27272 0.0 0.1 27032 1344 ? S Feb06 2:26 /usr/sbin/httpd -
www-data 29164 0.0 0.3 26296 3332 ? S Feb06 2:39 /usr/sbin/httpd -
www-data 19753 0.0 0.0 26300 952 ? S Feb06 2:14 /usr/sbin/httpd -
www-data 30495 0.0 0.2 26500 2504 ? S Feb06 2:21 /usr/sbin/httpd -
www-data 30496 0.0 0.9 26544 9468 ? S Feb06 2:19 /usr/sbin/httpd -
www-data 30497 0.0 0.5 26796 5504 ? S Feb06 2:22 /usr/sbin/httpd -
www-data 30526 0.0 0.0 26904 612 ? S Feb06 2:20 /usr/sbin/httpd -
www-data 8409 0.0 0.2 26520 2540 ? S Feb06 2:16 /usr/sbin/httpd -
www-data 8555 0.0 0.6 26744 7204 ? S Feb06 2:26 /usr/sbin/httpd -
www-data 26335 0.0 0.8 26544 8624 ? S Feb06 2:21 /usr/sbin/httpd -
www-data 26256 0.0 0.7 26564 7716 ? S Feb07 1:37 /usr/sbin/httpd -
root 18493 0.0 0.0 6388 0 ? SW 08:24 0:00 sshd: ayoung [pri
ayoung 18510 0.0 0.0 6432 304 ? S 08:24 0:02 sshd: ayoung@pts/
ayoung 18511 0.0 0.0 4604 0 pts/0 SW 08:24 0:00 -bash
root 18965 0.0 0.0 4312 0 pts/0 SW 08:26 0:00 su
root 18966 0.0 0.0 4624 672 pts/0 S 08:26 0:01 bash
named 25776 0.0 0.1 39060 1544 ? S 08:59 0:00 /usr/sbin/named -
root 13602 0.0 0.0 4200 108 pts/0 S 10:30 0:00 /bin/sh /usr/bin/
mysql 13657 0.4 0.2 15544 2144 pts/0 S 10:30 0:08 /usr/libexec/mysq
ayoung 20722 0.0 0.2 4044 2168 ? S 11:05 0:00 /usr/libexec/gcon
root 20988 0.0 0.0 2700 760 pts/0 R 11:06 0:00 ps aux
Nothing looks huge in the memory area. It hardly looks like the memory is being touched at all (7.2%).
I don't get it. Where did my RAM go? What else can I do to troubleshoot?
Thanks in advance,
-Alex
