I've got a system - RHEL 5.2 - that has 64Gb RAM and (only) 8Gb of Swap.

Both swap AND memory are at high levels of usage, which surprises me as we are runing basically only a database configured to use (maximum) of 48Gb.

I found this out through 'top' (only 7MB swap free). Running 'sar -B' gives the following averages :-

10:00:01 AM pgpgin/s pgpgout/s fault/s majflt/s
Average: 3516.60 1135.30 1418.30 0.06

The number of major faults is not very high, but there's evidently quite a bit of activity.


My question is will it be beneficial to increase the swap space on this box ?


(I can only add swap space in this situation via a swap file on slower disks, so I don't want to do something I'm stuck with - although it seems to me that if I enable a swap file I can disable it quite easily (anybody other ideas ?) via 'swapoff')

Anyody any ideas ?

Jack

Before increase swap, look for the reason of full memory.

Of course, there is the way to disable swap using 'swapoff /dev/swappartition' and enable it again 'swapon /dev/swappartition' as you mentioned.

Adding swap space would certainly increase the reliability of that system and probably increase the performance a little. If you have plenty of disk space, there should be no real downside to adding a big swap file.

It doesn't sound like it is important to worry about the speed of the disk containing that swap file. The swap space will be used to get stale anonymous data out of ram. Reading any of that back from swap space will likely be a very rare occurrence.

Since you have unexpected heavy memory use, you may want to investigate and get a better understanding of your system and maybe identify symptoms of a real problem before it becomes more serious. But such investigations can be difficult. I would add the extra swap space first, so the system is safer and maybe faster while you are investigating. Having enough swap space also may make the investigation easier, because it lets you identify which anonymous data is stale and it may let processes keep more of the non anonymous data they are actively using rather than soft fault thrashing that data through the cache (so you can more easily identify the heavy uses of non anonymous memory).

1 members found this post helpful.

My memory usage varies between 70%and 99% daily, but swap usage is constantly 99-100%.

The increase in memory usage coincides with increased user usage. The constant high rate of swap usage would suggest to me that the algorithm is wanting to do something like swap out aged/inactive processes all the time.

I've seen nothing dramatic in /var/log/messages.


Look at the following :-

[root@ct-on-db swapfile]# sar -r | more


12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
12:10:01 AM 174864 65902776 99.74 81484 61464412 0 8032460 100.00 1668
12:20:01 AM 175460 65902180 99.73 82384 61490980 0 8032460 100.00 1052
12:30:01 AM 173696 65903944 99.74 74516 61536168 0 8032460 100.00 928
12:40:01 AM 173252 65904388 99.74 74160 61543720 0 8032460 100.00 792
12:50:01 AM 171900 65905740 99.74 74964 61540960 0 8032460 100.00 780
01:00:01 AM 172456 65905184 99.74 75056 61538788 0 8032460 100.00 780
01:10:01 AM 177696 65899944 99.73 76404 61542460 0 8032460 100.00 844
01:20:01 AM 176132 65901508 99.73 68940 61557652 0 8032460 100.00 64
01:30:01 AM 178804 65898836 99.73 64916 61552204 0 8032460 100.00 56
01:40:01 AM 174240 65903400 99.74 63784 61539036 0 8032460 100.00 56
01:50:01 AM 167636 65910004 99.75 68452 61545268 0 8032460 100.00 56
02:00:01 AM 167400 65910240 99.75 70624 61549752 0 8032460 100.00 28
02:10:01 AM 177812 65899828 99.73 93044 61333204 0 8032460 100.00 224
02:20:01 AM 170564 65907076 99.74 59512 61211148 0 8032460 100.00 188
02:30:01 AM 172648 65904992 99.74 59156 61188800 0 8032460 100.00 188
02:40:01 AM 170432 65907208 99.74 59568 61204004 0 8032460 100.00 0
02:50:01 AM 168964 65908676 99.74 61340 61217476 0 8032460 100.00 0
03:00:01 AM 342040 65735600 99.48 41328 60275828 0 8032460 100.00 0
03:10:01 AM 2100364 63977276 96.82 42844 58827532 44 8032416 100.00 244
03:20:01 AM 396344 65681296 99.40 45600 61030108 44 8032416 100.00 244
03:30:01 AM 21589432 44488208 67.33 53900 40295924 56 8032404 100.00 328
03:40:01 AM 21576928 44500712 67.35 55236 40313088 88 8032372 100.00 552
03:50:01 AM 21562772 44514868 67.37 56972 40332560 88 8032372 100.00 552
04:00:01 AM 21551048 44526592 67.39 57880 40338632 88 8032372 100.00 552
04:10:01 AM 21451964 44625676 67.54 110616 40358620 96 8032364 100.00 608
04:20:01 AM 21457716 44619924 67.53 111496 40359844 96 8032364 100.00 608
04:30:01 AM 21451584 44626056 67.54 112384 40360876 96 8032364 100.00 608
04:40:01 AM 21449592 44628048 67.54 113272 40364532 96 8032364 100.00 608
04:50:01 AM 21437272 44640368 67.56 114120 40365404 96 8032364 100.00 608
05:00:01 AM 21444160 44633480 67.55 114908 40366284 96 8032364 100.00 608
05:10:01 AM 21439212 44638428 67.55 115708 40370160 96 8032364 100.00 608
05:20:01 AM 21437612 44640028 67.56 116528 40371068 96 8032364 100.00 608
05:30:01 AM 21434464 44643176 67.56 117336 40371992 100 8032360 100.00 604
05:40:01 AM 21430572 44647068 67.57 118140 40375136 100 8032360 100.00 604
05:50:01 AM 21437808 44639832 67.56 118968 40375672 100 8032360 100.00 604
06:00:01 AM 21435132 44642508 67.56 119780 40376848 100 8032360 100.00 604
06:10:01 AM 21429724 44647916 67.57 120620 40380412 100 8032360 100.00 604
06:20:01 AM 21428584 44649056 67.57 121384 40381120 100 8032360 100.00 604
06:30:01 AM 21426444 44651196 67.57 122224 40382164 104 8032356 100.00 632
06:40:01 AM 21422500 44655140 67.58 122940 40384896 104 8032356 100.00 632
06:50:01 AM 21422960 44654680 67.58 123620 40385588 104 8032356 100.00 632
07:00:01 AM 21410648 44666992 67.60 124376 40386700 104 8032356 100.00 632
07:10:01 AM 21415244 44662396 67.59 125168 40390560 104 8032356 100.00 632
07:20:01 AM 21413932 44663708 67.59 125908 40391232 104 8032356 100.00 632
07:30:01 AM 21418044 44659596 67.59 126652 40392476 104 8032356 100.00 632
07:40:01 AM 21407508 44670132 67.60 127412 40395608 104 8032356 100.00 632
07:50:01 AM 21403228 44674412 67.61 128120 40396356 104 8032356 100.00 632
08:00:01 AM 21400184 44677456 67.61 128900 40400020 104 8032356 100.00 632
08:10:01 AM 21402664 44674976 67.61 129652 40403840 104 8032356 100.00 632
08:20:01 AM 21400608 44677032 67.61 130404 40404700 104 8032356 100.00 632
08:30:01 AM 21074196 45003444 68.11 131792 40720384 208 8032252 100.00 1232
08:40:01 AM 21072244 45005396 68.11 132480 40724568 208 8032252 100.00 1232
08:50:01 AM 21067336 45010304 68.12 133140 40725252 208 8032252 100.00 1232
09:00:01 AM 21017932 45059708 68.19 133968 40777248 208 8032252 100.00 1232
09:10:01 AM 21013432 45064208 68.20 134668 40781712 228 8032232 100.00 1372
09:20:01 AM 21012808 45064832 68.20 135364 40782432 228 8032232 100.00 1372
09:30:01 AM 20308008 45769632 69.27 138912 41489048 276 8032184 100.00 1484
09:40:01 AM 20296064 45781576 69.28 139948 41494616 276 8032184 100.00 1484
09:50:01 AM 20315976 45761664 69.25 140660 41506212 276 8032184 100.00 1484
10:00:01 AM 20046040 46031600 69.66 142464 41781160 288 8032172 100.00 1504
10:10:01 AM 20004148 46073492 69.73 143428 41836000 288 8032172 100.00 1504
10:20:01 AM 19862484 46215156 69.94 145456 41908532 788 8031672 99.99 2860
10:30:01 AM 8106380 57971260 87.73 169272 53590840 1168 8031292 99.99 3664
10:40:01 AM 4772464 61305176 92.78 173244 56929400 1320 8031140 99.98 3960
10:50:01 AM 4743312 61334328 92.82 174140 56946800 1576 8030884 99.98 4504
11:00:01 AM 4700376 61377264 92.89 175764 56984556 2008 8030452 99.98 6248
11:10:01 AM 4569548 61508092 93.08 177020 57035476 2136 8030324 99.97 6952
11:20:01 AM 3508236 62569404 94.69 179000 58013240 2276 8030184 99.97 7260
11:30:01 AM 2970388 63107252 95.50 180652 58445012 2380 8030080 99.97 7348
11:40:01 AM 2588436 63489204 96.08 181620 58722636 2612 8029848 99.97 7660
11:50:01 AM 2244856 63832784 96.60 182740 58984344 3748 8028712 99.95 10268
12:00:01 PM 2019972 64057668 96.94 183892 59203528 4284 8028176 99.95 12580
12:10:01 PM 2090324 63987316 96.84 185496 59382744 4484 8027976 99.94 13692
12:20:01 PM 1737688 64339952 97.37 186912 59645852 5288 8027172 99.93 16984

Johnsfine/Anybody else,

Is there any risk in creating a swap file on a software raid device (/dev/md0) ? I assume that when looking at /etc/fstab on reboot, Linux will resolve all filesystems before invoking the use of a swap file ?

Jack

The constant high swap space usage indicates you don't have enough swap space.

Linux does not swap out inactive processes, it swaps out inactive anonymous pages from processes, regardless of whether the processes are active or inactive.

Many server processes uses significant anonymous memory during startup that is then untouched until the process shuts down. Those pages should be swapped out.

If that is typical, you have no problem. The key number is kbcached. That remained above half your total memory for that whole set of stats. With kbcached that high, the performance improvement from increasing swap size would be too tiny to measure.
With some very unusual memory allocation patterns by processes, it is possible that those processes could fail due to lack of swap space even with kbcached that high, but it is extremely unlikely.
I would allocate more swap space anyway. I really hate to see swap mostly used in normal operation. But there may not be a serious need to allocate more swap.

That is far less significant than you think. 99% memory use is fine as long as cache is the major user of memory. Read this:
http://www.linuxatemyram.com/

Lots of people allocate swap files on software RAID devices. I've never done so myself, so I don't know exactly what you need in /etc/fstab and other startup files to get everything to happen in the correct sequence. I still wouldn't really call that a "risk". If things happen in the wrong sequence on startup the extra swap space simply won't get added. That is no worse than not having it set up in the first place.

Just remember after the next reboot to check to see if your extra swap space was added properly during startup. If not, you can manually add it, try fixing whatever the startup sequence issue might be and check it again the time after that.

1 members found this post helpful.

johnsfine

Thank you. I added the swap file of 8GB last night and everything seems fine.
Swap use has increased to about 14Gb in total and seems to have stablised - it seems to have had the effect of dropping total memory usage between 60- 80% max, although I want to see what happens over the rest of the day.

Jack

I expect memory use will get back up to 99%. Once everything settles out, the extra swap space will be allowing the system to use more cache space, which should result in a very tiny improvement in performance.

Remember, all that cache space is only used as long as the system doesn't need that memory for anything else.

Assuming you have a RHEL subscription, here are RH's guidelines https://access.redhat.com/kb/docs/DOC-15252