Transparent Huge Pages usage on RHEL 6

LinuxLover · 10-01-2014, 11:27 PM

Hi geeks,
We have many RHEL servers being used only for BIND DNS no other application on these servers.Mainly two RHEL version are being used RHEL 5&6.

It is observed that memory utilization on RHEL 6 machine remains high and RHEL 6 systems keep on increasing the memory gradually whereas we have set the Bind max cache size at 2GB.
After lot of investigation and google searching it is discovered that running process memory utilization is not high like check the below stats

Code:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          7857       7574        283          0        483       2731
-/+ buffers/cache:       4359       3498
Swap:         8191         13       8178

Here system is using 4359MB acutal RAM.

If I check the RSS memory usage of all running process

Code:

]$ ps -e --format rss  | awk 'BEGIN{c=0} {c+=$1} END{print c/1024}'
1917.84

It is less the 2GB.

If we check the kernel memory usage ( Slab)

Code:

]$ cat /proc/meminfo | grep Slab
Slab:            1024312 kB

It is around 1GB

By consulting RHEL 6 documentation , it is found they have enable by default Transparent Huge Pages (AnonHugePages). So if we check the

Code:

]$ cat /proc/meminfo |grep AnonHugePages
AnonHugePages:   1728512 kB

Now I have two question
1- Does these AnonHugePages memory utilization is separate or it is included under used memory of processes ( RSS memory of all running proccess in system)

2- If our Total memory in system 8GB or less , should we disable the AnonHugePages or keep it enable ?

After few weeks or months system start using swap memory and performance degraded.So we have to reboot the machine.

syg00 · 10-02-2014, 12:51 AM

Spin through /proc/*/smaps looking for "Huge"; there'll be a *lot* of them - sum them per pid in a similar awk to above. Or just look at the pid for bind.
That'll tell you who has them allocated.

The ony way I can see this being a problem is if the process asks for a page, writes something (small) then allocates a new page, and gets a huge page allocated each time.
As for turning them off - if it causes a problem, sure. But it's your system not mine ....

Can be done on the fly, or at boot.

LinuxLover · 10-02-2014, 02:27 AM

Dear syg00, Thanks for your reply.

If I sum up the AnonHugePages of each process it is exactly the same as system depicting in meminfo look below

Code:

# grep AnonHugePages /proc/*/smaps | awk '{print $2}' |grep -v "^0" |  awk '{sum += $1;} END {print sum " KB" ;}'
1728512 KB

and

Code:

# cat /proc/meminfo |grep AnonHugePages
AnonHugePages:   1728512 kB

If I check only the BIND process AnonHugePages it is using more then 99%

Code:

]# grep AnonHugePages /proc/6839/smaps | awk '{print $2}' |grep -v "^0" |  awk '{sum += $1;} END {print sum " KB" ;}'
1724416 KB

From this I am taking the impression that these running process are using AnonHugePages but it is not included in RSS memory of process (ps aux ).Where as Free command count these AnonHugePage under actual used memory of system. Am I right ?

Still unable to decided should I turn it off or not on system having less then 8GB memory.

LinuxLover · 10-06-2014, 10:13 PM

Still looking for some more feedback