[SOLVED] NUMA clarifications on a Supermicro built

lpallard · 11-11-2013, 04:54 PM

Hi all!

I have initially posted this on another forum, but got no replies. Searching here also led to not much. Google also offers noting significant, unless I completely missed the boat, which is possible...

So my server has been running for about 3 months now with a slackware14-64 and kernel 3.2.45 migration from an older setup. The current running kernel is configured with the following flags:

Code:

CONFIG_X86_64_SMP=y
CONFIG_SMP=y
CONFIG_NR_CPUS=32

And returns this when probed about type and version:

Code:

root@lhost2:~# uname -a
Linux lhost2 3.2.45 #2 SMP Fri May 31 20:14:55 CDT 2013 x86_64 AMD Opteron(tm) Processor 4334 AuthenticAMD GNU/Linux

Now to the question: when building this machine, we were made aware that there was such a thing as NUMA and UMA... Reading about the topic on Google and provided I understand correctly, NUMA is an architecture where in multi-socket machines, memory populated together with a CPU consists of a node. A certain chip (CPU) would be allowed to access its local memory bank(s) faster than accessing remote memory bank(s) located on another node. Hardware wise, the H8DCL-IF motherboard I am using complies with this. RAM is also populated as per Supermicros guidelines, that is, with 4 identical sticks (a kit of 4X 16GB), 2 on each node on channels A of each chips.

The BIOS is currently set to "Enable NUMA", has bank and channel interleaving ON and has Node interleaving OFF. In my world, that means NUMA is fully activated, yet when I run the utility "numactl --hardware" it causes a segmentation fault and crashes. According to the numactl maillist, this would be because my platform doesnt support NUMA (or is not properly configured). This is consistent with running

Code:

numactl --show

which returns

Code:

physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 
No NUMA support available on this system.

The running kernel doesnt seems to support (with its current config) NUMA as I could only find this flag in the kernel's running config:

Code:

# CONFIG_NUMA is not set

So the questions:

1-Should I make NUMA work or will I suffer a performance penalty?

2-Do I need to recompile the kernel with NUMA activated? Threads on other Linux forums seemed to indicate that the modern kernels were automatically NUMA aware and did not require special flags...

3-Is my BIOS config OK? (Node interleave OFF, Bank interleave ON, Channel interleave ON, NUMA enabled)

4-The numactl page (https://www.sharcnet.ca/help/index.php/Using_numactl) talks about NUMA:

Quote:

This is in contrast to symmetric multiprocessing or SMP architecture in which each processor is connected to the whole of main memory at uniform speed.

I am currently using a SMP kernel.

According to their statement, that would mean that my current config uses a UMA architecture... Is it why numactl seems to think that NUMA is not available?

Thanks for any insight or help!

Pearlseattle · 11-12-2013, 02:50 PM

Hi

I think that the key is

Quote:

# CONFIG_NUMA is not set

If it's not set, then it's not active and it won't be used, no?

Quote:

2-Do I need to recompile the kernel with NUMA activated? Threads on other Linux forums seemed to indicate that the modern kernels were automatically NUMA aware and did not require special flags...

I would say that your system won't be numa-aware if numa is inactive in your kernel.
Ideally it should look similar to this:

Quote:

grep -i numa .config
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
# CONFIG_NUMA_BALANCING is not set
CONFIG_NUMA=y
# CONFIG_AMD_NUMA is not set
CONFIG_X86_64_ACPI_NUMA=y
# CONFIG_NUMA_EMU is not set
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ACPI_NUMA=y

Quote:

1-Should I make NUMA work or will I suffer a performance penalty?

The 1000000000$ question - I read all kind of opinions about it, ranging between the two extremes. I really mean it: found 0 sources which kind of agreed.
Maybe all the operations that are performed by your server can be done by using your L1-cache in which case numa would just be ballast, but maybe not?
The best way for you to decide is most probably to write your own program (or run a simulation) which simulates the typical server load (or a worst-case-scenario) you (could) have and compare the runtimes.

What do you think?

lpallard · 11-12-2013, 09:14 PM

Quote:

If it's not set, then it's not active and it won't be used, no?

Absolutely! I am just surprised there are so many flags to build NUMA into the kernel.... My search on google had not indicated what flags had to be inserted in the config file for NUMA to be built in the kernel... Good to know!

Basically, I understand that the following type/kind of statement indicates that the associated flags are simply not defined rather than deactivated... Is that correct?

Code:

# CONFIG_WHATEVER_KERNEL_OPTION is not set

If true, then that would explain why NUMA is not present in my running kernel: because the flags are not defined (not set) instead of being explicitly deactivated (=n)

Quote:

Ideally it should look similar to this:

Quote:

grep -i numa .config
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANTS_PROT_NUMA_PROT_NONE=y
# CONFIG_NUMA_BALANCING is not set
CONFIG_NUMA=y
# CONFIG_AMD_NUMA is not set
CONFIG_X86_64_ACPI_NUMA=y
# CONFIG_NUMA_EMU is not set
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ACPI_NUMA=y

Where can I get the full list of possible flags??? I was expecting only 1 or 2 flags but I see that there are at least 6 active in your config, and at least 3 more because of the "not set" ones in your config...

In regrds to the performance, I agree with you: there is no consensus in the field... What I have found are opinions ranging from "don't touch that" to "a 200% improvement"...

I wonder if, instead of writing my own application, I was performing CPU/memory intensive work on the server, then trying with a NUMA-aware kernel, and comparing the same task's runtimes?

Pearlseattle · 11-14-2013, 12:01 PM

Hi

Quote:

Basically, I understand that the following type/kind of statement indicates that the associated flags are simply not defined rather than deactivated... Is that correct?

Well, yes and no: by being not defined, they're deactivated

There are many other kernel options that if activated, generated additional flags in the kernel config file that would otherwise not be present with a deactivated option. I'm not an expert, but I think that those additional flags don't appear until the first time you activate the main option and save the config => then even if you deactivate the option those other flags will still be present, but with the "N" value (but I might be wrong - I never looked in detail into this...).

Quote:

Where can I get the full list of possible flags???

Probably the only way is to start up the kernel config menu (I mean the "make menuconfig") and search for the "numa" string... .
Options will vary depending on the kernel version and the potential customizations that your distribution will have implemented.

Quote:

I wonder if, instead of writing my own application, I was performing CPU/memory intensive work on the server, then trying with a NUMA-aware kernel, and comparing the same task's runtimes?

Yes, anything you want, as long as 1) it represents a typical load you have (so that the results are useful for your scope), 2) can be executed again and again following the same pattern (so that you can compare the results) and 3) there are no external influences to the tests (e.g. daemons running in the background or whatever putting unexpected load during only some of the tests).
So, in the end it's the usual performance test

At that point you'll be able to compare the test results and take a correct decision, regardless of what everybody else says.

Good luck!

lpallard · 11-21-2013, 07:26 PM

OK I am not sure if this is related to the topic here but I have got strange messages in the server's dmesg and a quick google search didnt point to anything uyseful....

Code:

[1621434.026513] The scan_unevictable_pages sysctl/node-interface has been disabled for lack of a legitimate use case.  If you have one, please send an email to linux-mm@kvack.org.
[1621434.033593] process `sysctl' is using deprecated sysctl (syscall) net.ipv6.neigh.default.retrans_time; Use net.ipv6.neigh.default.retrans_time_ms instead.

What in the name of God are those???

All I got searching on Google were bug reports for the kernel.... Did I hit a kernel bug?!

Pearlseattle · 11-23-2013, 08:14 AM

No clue.
Maybe some mismatch between kernel and kernel modules, or kernel (modules) and userspace programs... .