Memory Page Out Issues - RHEL 5.3 x86_64
 

Hi LQ Community,

I have an issue with two RHEL 5.3 x86_64 servers running Oracle RAC 10.2.0.

Both the servers are HP ProLiant DL380 G7 with 36GB of RAM.

At random times (a few times a day) our monitoring solution reports page outs on both the servers as high as 6000. This always happens at the same time on both servers. The alert then disappears after about 5 minutes and reports page outs are almost non existent again.

top shows that the servers aren't swapping heavily at all and there is generally around 5-6GB of free RAM on both servers.

I was wondering if there was any way of pin pointing what process is actually causing the page outs so I can address the issue.

I have tried looking at sar and vmstat but from what I can see they don't provide this functionality. However, I may have missed/not understood something.

If anyone has any suggestions on how to investigate this issue further I would greatly appreciate it.

If any other information is required to provide some advice please let me know and I will post the information.

Thanks in advance.

Cheers

Only thing I know is systemtap.Please refer to the below link and let me know in case of any issue because I know its really complicated to setup :)

https://access.redhat.com/knowledge/...ndex.html.html

Script you need to run once its setup --> traceio.stp

Thanks
Prashant

Forget to mention it will give you output like this which I think will pin-point you which process is taking memory..

2664 Xorg r: 2 MiB w: 0 B
3311 firefox r: 14 KiB w: 303 KiB
2988 pulseaudio r: 36 KiB w: 36 KiB
3653 gnome-terminal r: 21 KiB w: 28 KiB
9479 stapio r: 18 KiB w: 20 KiB
1944 dbus-daemon r: 12 KiB w: 0 B
2121 hald-addon-inpu r: 9 KiB w: 0 B
2457 crond r: 7 KiB w: 0 B
1859 cgrulesengd r: 6 KiB w: 0 B
2944 metacity r: 6 KiB w: 0 B

Hi lakhera2010,
Thanks for your suggestion.

I am back in the office today so I will check it out later this afternoon.

I will let you know how I go.

Cheers

Hi lakhera2010,
Managed to track it down to being an oracle process with systemtap.

The DBA's here advised that they can do some tuning on the database side to utilise more ram so that the page outs don't occur.

Thanks for your help with this, much appreciated.

Cheers