Disabling THP

umoralwar · 10-27-2015, 05:13 PM

we have system running on below patcset

uname -r
2.6.32-504.8.1.el6.x86_64

we have disabled thp on this host

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]
cat /sys/kernel/mm/redhat_transparent_hugepage/defrag
always madvise [never]
cat /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag
[yes] no

however checking compact counter I see lot of stalls

cat /proc/vmstat | grep compact
compact_blocks_moved 1123350
compact_pages_moved 34307235
compact_pagemigrate_failed 0
compact_stall 648
compact_fail 29
compact_success 619

just want to understand if THP is disabled what else is triggering compact_page_moved and compact_stall

Demosa · 10-28-2015, 10:36 AM

To help people looking at this/search tags, THP is Transparent Huge Pages

Anyways,
So, you're running RHEL 6.6 (or a RHEL 6 derivative) on 64 bit (THP does not exist on 32 bit systems)

From https://www.kernel.org/doc/Documenta.../transhuge.txt

Quote:

As the system ages, allocating huge pages may be expensive as the
system uses memory compaction to copy data around memory to free a
huge page for use. There are some counters in /proc/vmstat to help
monitor this overhead.

compact_stall is incremented every time a process stalls to run
memory compaction so that a huge page is free for use.

compact_success is incremented if the system compacted memory and
freed a huge page for use.

compact_fail is incremented if the system tries to compact memory
but failed.

compact_pages_moved is incremented each time a page is moved. If
this value is increasing rapidly, it implies that the system
is copying a lot of data to satisfy the huge page allocation.
It is possible that the cost of copying exceeds any savings
from reduced TLB misses.

compact_pagemigrate_failed is incremented when the underlying mechanism
for moving a page failed.

compact_blocks_moved is incremented each time memory compaction examines
a huge page aligned range of pages.

So, with those numbers, it does seem that hugepages is enabled and the memory usage is causing some overhead processes to kick off.

Did you append "transparent_hugepage=never" in grub.conf?
Also, run the following and post the results

grep HugePages /proc/meminfo
grep DirectMap /proc/meminfo
service tuned status
service ktune status
chkconfig --list | egrep "tuned|ktune"

umoralwar · 10-28-2015, 06:13 PM

Hello Demosa,

Thanks for the reply.

These are DB hosts and we have Oracle database running with huge pages configured for SGA my understanding is THP and they are not same.
(sorry for my limited understanding I am still learning.)

Question is turning on huge pages for Oracle will also update compact counters?

To answer , yes we have appended "transparent_hugepage=never" in grub.conf

grep HugePages /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 400000
HugePages_Free: 293845
HugePages_Rsvd: 16727
HugePages_Surp: 0

grep DirectMap /proc/meminfo
DirectMap4k: 4212 kB
DirectMap2M: 2082816 kB
DirectMap1G: 2145386496 kB

service tuned status
tuned: unrecognized service

service ktune status
ktune: unrecognized service

chkconfig --list | egrep "tuned|ktune" <- did not return which indicates we don't have above service running.

Demosa · 10-29-2015, 09:02 AM

Whoa.... okay, so you've got 800 GB of hugepages RAM on this server? And more than 2 TB of RAM? Might need to hand this off to a professional, or at least reference the professional documentation (With a server like that, I'd imagine you'd have a RHEL or Oracle Linux license, and I know both of them have Oracle configuration and tuning guides for hugepages). If this isn't accurate, then there's some memory configuration stuff that isn't right.

But to try to dig a bit more into what might be going on...
AnonHugePages 0kb indicates to me that THP is correctly disabled
HugePages_Total 400000 indicates that explicit hugepages are on, and that there are 400,000 of them
There are some virtual memory parameters to investigate, to ensure the server is well tuned for an oracle DB that would reduce/eliminate compact hits.

What I forgot to have you do was verify the size of your hugepages, just to make sure they are in fact 2 MB

Code:

grep Hugepagesize /proc/meminfo

umoralwar · 10-29-2015, 09:58 AM

Hello Demosa,

Yes huge page size is of 2MB.