LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Other *NIX Forums > AIX
User Name
Password
AIX This forum is for the discussion of IBM AIX.
eserver and other IBM related questions are also on topic.

Notices

Reply
 
LinkBack Search this Thread
Old 09-27-2012, 04:14 AM   #1
jsreeni
LQ Newbie
 
Registered: Sep 2012
Posts: 1

Rep: Reputation: Disabled
Paging space fills up after few days


Hi Team,

I am a newbie to AIX and I am currently facing a problem where the AIX server fills the entire memory and paging space every few days. On a daily basis, this seems to be fine but after running for a month or so then automatically, there is a sudden burst of usage which fills up both the real and paging space memory and we have to reboot the system in order to bring back the server.

The server contains applications running on Glassfish and a oracle Database running on the server.

>lsattr -El sys0 | grep realmem
realmem 25165824 Amount of usable physical memory in Kbytes False

>lsps -s
Total Paging Space Percent Used
32000MB 2%

>vmstat -v
6291456 memory pages
6080672 lruable pages
42674 free pages
1 memory pools
709454 pinned pages
80.0 maxpin percentage
3.0 minperm percentage
80.0 maxperm percentage
71.4 numperm percentage
4346914 file pages
0.0 compressed percentage
0 compressed pages
71.4 numclient percentage
80.0 maxclient percentage
4346914 client pages
0 remote pageouts scheduled
0 pending disk I/Os blocked with no pbuf
840514 paging space I/Os blocked with no psbuf
2228 filesystem I/Os blocked with no fsbuf
0 client filesystem I/Os blocked with no fsbuf
577 external pager filesystem I/Os blocked with no fsbuf
30.2 percentage of memory used for computational pages

>vmstat 1 10
System configuration: lcpu=4 mem=24576MB ent=0.50

kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
1 0 1931190 46520 0 0 0 0 0 0 5 498 533 1 3 96 0 0.04 7.2
10 0 1933117 44593 0 0 0 0 0 0 32 27994 1232 22 33 45 1 0.36 72.6
2 0 1932210 45500 0 0 0 0 0 0 133 22509 1324 23 31 46 1 0.38 75.0
1 0 1931557 46153 0 0 0 0 0 0 8 4592 569 11 6 83 0 0.14 28.3
2 0 1933507 44201 0 0 0 0 0 0 44 148932 877 53 15 32 0 0.89 177.6
1 0 1933826 43882 0 0 0 0 0 0 14 8407 736 25 13 62 0 0.28 55.8
1 0 1931630 46078 0 0 0 0 0 0 11 14088 849 18 16 65 0 0.26 52.6
1 0 1931231 46477 0 0 0 0 0 0 6 3953 574 11 6 83 0 0.14 28.4
1 0 1931231 46477 0 0 0 0 0 0 2 366 533 1 2 97 0 0.03 5.3
2 0 1923533 54175 0 0 0 0 0 0 5 776 532 1 5 94 0 0.05 10.3
10 0 1924744 52964 0 0 0 0 0 0 40 42731 1425 36 43 20 0 0.51 101.8
1 0 1925282 52425 0 0 0 0 0 0 31 33646 1718 46 34 20 0 0.49 97.2
1 0 1924105 53602 0 0 0 0 0 0 19 60401 3753 34 37 29 0 1.00 199.8
2 0 1923640 54067 0 0 0 0 0 0 4 63341 3821 31 39 30 0 1.00 199.6
0 0 1923733 53975 0 0 0 0 0 0 6 5845 757 13 7 79 0 0.17 33.2
0 0 1923733 53975 0 0 0 0 0 0 6 347 543 0 3 97 0 0.03 6.3
0 0 1923733 53975 0 0 0 0 0 0 6 706 544 1 3 96 0 0.03 6.5
0 0 1923733 53975 0 0 0 0 0 0 6 700 552 1 3 96 0 0.03 6.9
0 0 1923733 53975 0 0 0 0 0 0 4 581 558 1 3 97 0 0.03 6.0
0 0 1923733 53974 0 0 0 0 0 0 8 587 546 1 2 97 0 0.02 4.8


Any help is appreciated with respect to the paging space issue.
 
Old 09-27-2012, 08:42 AM   #2
crabboy
Moderator
 
Registered: Feb 2001
Location: Atlanta, GA
Distribution: Slackware
Posts: 1,822

Rep: Reputation: 120Reputation: 120
Host much memory is allocated for Glassfish to use and how many java VM instances are set to run from it?

Also, what is the memory consumption for Oracle?

It sounds like these applications are consuming all 24 gigs of your RAM and then it eats your paging space. Do you run nmon on the machine? that can produce a detailed report of memory and CPU usage over time.

Can you post the top part of the prtconf command, just until the Volume Group info? Probably best to remove the serial number from the output.
 
Old 09-27-2012, 07:45 PM   #3
xj-linux
Member
 
Registered: Feb 2006
Location: Minnesota
Distribution: SUSE - AIX
Posts: 30

Rep: Reputation: 2
Oracle is a pig if left unchecked. Check, or have your DBA check, the memory parameters for your Oracle instance(s). Basically, it will eat up as much memory as you have available unless you set a cap on it. Check the SGA settings - max, size, etc... Left to it's own devices, it will eat a server out of house and home, especially if you have redo logs, and various housekeeping tasks that run intermittently. It can chew through RAM and then into paging space. We have a large SAP ERP instance and have Oracle set to 60% or 70% of available memory on a 72GB LPAR with 20GB of paging space (other stuff is running on there as well as the operating system that need memory). Java can be as bad, but I stay away from it whenever possible so I'm of no help there.
 
Old 10-03-2012, 03:52 AM   #4
cliffordw
Member
 
Registered: Jan 2012
Location: South Africa
Posts: 247

Rep: Reputation: 106Reputation: 106
Hi there,

Does the memory usage increase gradually, or stay flat and then suddently go haywire? This ight help you understand whether it is a gradual memory leak or just one job that misbehaves.

I'd recommend that you capture the output of the commands you have already mentioned above on a regular basis. You might also want to add "svmon -G" and "svmon -u" to that list. The latter will show you how much memory each user is using, and help you pinpoint the culprit.

Good luck!
 
Old 10-10-2012, 03:11 PM   #5
scheerchrs
LQ Newbie
 
Registered: Oct 2012
Posts: 3

Rep: Reputation: Disabled
Good amount of tips from previous posters; also, Oracle app and db servers tend to incorporate significant weekly/monthly/quarterly batch job processing; check with the dba's to see if the memory leak/utilization issue corresponds with a batch or other db job that runs; that will give a good clue, in my experience, to what's running up your memory utilization at "given intervals".

Quick fix: add paging space big time. (triple it?) (*DYNAMIC AND IMMEDIATE, NO REBOOT NEEDED). That should at least give you breathing room to root cause whatever's hog-tieing up physical memory. Follow through with the above suggestions on system kernel tuning for max processes per user, semaphore settings, etc for Oracle...Oracle is a spoiled brat when it comes to kernel tuning needs.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Increase paging space iregk AIX 3 12-01-2010 11:08 PM
[SOLVED] Difference between Swap and Paging space ?? vikas027 Linux - General 3 09-02-2008 12:00 AM
System lags and eats a lot of swap space over a few days Kremit Linux - General 4 06-27-2005 12:30 PM
AIX server running out of paging space ravikumarG AIX 8 05-16-2005 09:23 AM
paging space jnc Programming 0 09-12-2001 11:03 AM


All times are GMT -5. The time now is 10:55 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration