Hi Team,

I am a newbie to AIX and I am currently facing a problem where the AIX server fills the entire memory and paging space every few days. On a daily basis, this seems to be fine but after running for a month or so then automatically, there is a sudden burst of usage which fills up both the real and paging space memory and we have to reboot the system in order to bring back the server.

The server contains applications running on Glassfish and a oracle Database running on the server.

>lsattr -El sys0 | grep realmem
realmem 25165824 Amount of usable physical memory in Kbytes False

>lsps -s
Total Paging Space Percent Used
32000MB 2%

>vmstat -v
6291456 memory pages
6080672 lruable pages
42674 free pages
1 memory pools
709454 pinned pages
80.0 maxpin percentage
3.0 minperm percentage
80.0 maxperm percentage
71.4 numperm percentage
4346914 file pages
0.0 compressed percentage
0 compressed pages
71.4 numclient percentage
80.0 maxclient percentage
4346914 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
840514 paging space I/Os blocked with no psbuf
2228 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
577 external pager filesystem I/Os blocked with no fsbuf
30.2 percentage of memory used for computational pages

>vmstat 1 10
System configuration: lcpu=4 mem=24576MB ent=0.50

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
1 0 1931190 46520 0 0 0 0 0 0 5 498 533 1 3 96 0 0.04 7.2
10 0 1933117 44593 0 0 0 0 0 0 32 27994 1232 22 33 45 1 0.36 72.6
2 0 1932210 45500 0 0 0 0 0 0 133 22509 1324 23 31 46 1 0.38 75.0
1 0 1931557 46153 0 0 0 0 0 0 8 4592 569 11 6 83 0 0.14 28.3
2 0 1933507 44201 0 0 0 0 0 0 44 148932 877 53 15 32 0 0.89 177.6
1 0 1933826 43882 0 0 0 0 0 0 14 8407 736 25 13 62 0 0.28 55.8
1 0 1931630 46078 0 0 0 0 0 0 11 14088 849 18 16 65 0 0.26 52.6
1 0 1931231 46477 0 0 0 0 0 0 6 3953 574 11 6 83 0 0.14 28.4
1 0 1931231 46477 0 0 0 0 0 0 2 366 533 1 2 97 0 0.03 5.3
2 0 1923533 54175 0 0 0 0 0 0 5 776 532 1 5 94 0 0.05 10.3
10 0 1924744 52964 0 0 0 0 0 0 40 42731 1425 36 43 20 0 0.51 101.8
1 0 1925282 52425 0 0 0 0 0 0 31 33646 1718 46 34 20 0 0.49 97.2
1 0 1924105 53602 0 0 0 0 0 0 19 60401 3753 34 37 29 0 1.00 199.8
2 0 1923640 54067 0 0 0 0 0 0 4 63341 3821 31 39 30 0 1.00 199.6
0 0 1923733 53975 0 0 0 0 0 0 6 5845 757 13 7 79 0 0.17 33.2
0 0 1923733 53975 0 0 0 0 0 0 6 347 543 0 3 97 0 0.03 6.3
0 0 1923733 53975 0 0 0 0 0 0 6 706 544 1 3 96 0 0.03 6.5
0 0 1923733 53975 0 0 0 0 0 0 6 700 552 1 3 96 0 0.03 6.9
0 0 1923733 53975 0 0 0 0 0 0 4 581 558 1 3 97 0 0.03 6.0
0 0 1923733 53974 0 0 0 0 0 0 8 587 546 1 2 97 0 0.02 4.8


Any help is appreciated with respect to the paging space issue.

Host much memory is allocated for Glassfish to use and how many java VM instances are set to run from it?

Also, what is the memory consumption for Oracle?

It sounds like these applications are consuming all 24 gigs of your RAM and then it eats your paging space. Do you run nmon on the machine? that can produce a detailed report of memory and CPU usage over time.

Can you post the top part of the prtconf command, just until the Volume Group info? Probably best to remove the serial number from the output.

Oracle is a pig if left unchecked. Check, or have your DBA check, the memory parameters for your Oracle instance(s). Basically, it will eat up as much memory as you have available unless you set a cap on it. Check the SGA settings - max, size, etc... Left to it's own devices, it will eat a server out of house and home, especially if you have redo logs, and various housekeeping tasks that run intermittently. It can chew through RAM and then into paging space. We have a large SAP ERP instance and have Oracle set to 60% or 70% of available memory on a 72GB LPAR with 20GB of paging space (other stuff is running on there as well as the operating system that need memory). Java can be as bad, but I stay away from it whenever possible so I'm of no help there.

Hi there,

Does the memory usage increase gradually, or stay flat and then suddently go haywire? This ight help you understand whether it is a gradual memory leak or just one job that misbehaves.

I'd recommend that you capture the output of the commands you have already mentioned above on a regular basis. You might also want to add "svmon -G" and "svmon -u" to that list. The latter will show you how much memory each user is using, and help you pinpoint the culprit.

Good luck!

Good amount of tips from previous posters; also, Oracle app and db servers tend to incorporate significant weekly/monthly/quarterly batch job processing; check with the dba's to see if the memory leak/utilization issue corresponds with a batch or other db job that runs; that will give a good clue, in my experience, to what's running up your memory utilization at "given intervals".

Quick fix: add paging space big time. (triple it?) (*DYNAMIC AND IMMEDIATE, NO REBOOT NEEDED). That should at least give you breathing room to root cause whatever's hog-tieing up physical memory. Follow through with the above suggestions on system kernel tuning for max processes per user, semaphore settings, etc for Oracle...Oracle is a spoiled brat when it comes to kernel tuning needs.