Cannot fork after running CPIO backup
Having a major problem with RedHat AS 3 running on an HP DL760 G2 with 8 CPUs. Every night I do a CPIO file backup to tape. Before the CPIO starts I generally have around 50%+ free memory. 5g out of about 10g is being used. After the CPIO starts a sar -r shows the free memory just getting sucked down until 99% of it is used. When the CPIO finishes free memory is still at 99% used and shortly after that the box starts generating these cannot fork messages. Once I start seeing those cannot fork messages the box becomes unstable. Once it becomes unstable my only recourse is to power cycle the box as I can't run any commands at all. I spoke to HP who contacted RedHat and they first told me to upgrade to the latest SMP kernel so I did and ran a complete up2date with the same results. I was then told I should switch to the hugemem kernel because I had so much ram but that didn't help either. When that didn't work I was told that even though a sar -r showed all the memory as being used it wasn't in fact used and was available to the kernel if it needed it. Now if that were the case how come the box becomes unstable afterwards? Rather than doing a CPIO I also tried a tar to tape with the same results so it almost seems to be IO related. Below is a sar -r output that I captured right before the CPIO that I ran at 12:35. You can see as soon as the CPIO runs the free memory just gets sucked away. I've been working on this problem since Dec with HP and RedHat and we are still no where. Anyone else have any ideas?
12:00:00 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
12:00:00 AM 3333444 3607192 51.97 32032 2916680 2097112 0 0.00 0
12:10:00 AM 3330356 3610280 52.02 32388 2916964 2097112 0 0.00 0
12:20:00 AM 3328428 3612208 52.04 32720 2917920 2097112 0 0.00 0
12:30:00 AM 3255100 3685536 53.10 34188 2990532 2097112 0 0.00 0
12:40:01 AM 649532 6291104 90.64 118116 5397260 2097112 0 0.00 0
12:50:00 AM 22792 6917844 99.67 172684 5837764 2097112 0 0.00 0
01:00:00 AM 18136 6922500 99.74 187876 5827396 2097112 0 0.00 0
01:10:00 AM 18148 6922488 99.74 64328 5947684 2097112 0 0.00 0
01:20:01 AM 47440 6893196 99.32 7780 5973228 2097112 0 0.00 0
01:30:00 AM 44592 6896044 99.36 8996 5975408 2097112 0 0.00 0
01:40:00 AM 37124 6903512 99.47 10216 5980832 2097112 0 0.00 0
01:50:01 AM 49080 6891556 99.29 11756 5958904 2097112 0 0.00 0
02:00:00 AM 49148 6891488 99.29 12836 5958992 2097112 0 0.00 0
|
