Hello,

I'm benchmarking our new Supermicro servers and i noticed poor performances of the memset and memcpy fonctions

The config : 2x E5-2420, 12 x 8Gb DDR3 ECC reg 12800, SSD , debian stable amd64 up to date

Stream results are really good :
Function Best Rate MB/s Avg time Min time Max time
Copy: 35890.9 0.004463 0.004458 0.004470
Scale: 35715.2 0.004493 0.004480 0.004531
Add: 36412.8 0.006597 0.006591 0.006601
Triad: 36535.7 0.006580 0.006569 0.006588

but bandwidth-0.32 is giving me poor library performances :
Library: memset 3991.9 MB/s
Library: memcpy 2373.5 MB/s


If i compare with our development server Dell R210 server ( 1 x E31240, 4 x 4 GB DDR3 1333, sata 10k, debian stable amd64 up to date),
stream is much slower :
Function Best Rate MB/s Avg time Min time Max time
Copy: 11862.3 0.023112 0.013488 0.043449
Scale: 11829.9 0.022943 0.013525 0.034822
Add: 13162.2 0.024458 0.018234 0.034396
Triad: 13263.3 0.026572 0.018095 0.036060


but, in use, memset and memcpy are really faster :
Library: memset 15411.6 MB/s
Library: memcpy 7925.9 MB/s


Is there something i can do ?

What could be the reason of the poor performances of Supermicro ?

Thank's a lot for help !

have you tried another tool also? there is a comment on http://zsmith.co/bandwidth.html: Be nice On Linux, I recommend using nice -n -2 when running bandwidth. The kernel may attempt to throttle the process otherwise.
Have you checked it?

Thanks for the help.

Tryed with nice but changed nothing

I tested with mbw :
Long uses 8 bytes. Allocating 2*134217728 elements = 2147483648 bytes of memory.
Using 262144 bytes as blocks for memcpy block copy test.
Getting down to business... Doing 1 runs per test.
0 Method: MEMCPY Elapsed: 0.24884 MiB: 1024.00000 Copy: 4115.061 MiB/s
AVG Method: MEMCPY Elapsed: 0.24884 MiB: 1024.00000 Copy: 4115.061 MiB/s
0 Method: DUMB Elapsed: 0.43245 MiB: 1024.00000 Copy: 2367.876 MiB/s
AVG Method: DUMB Elapsed: 0.43245 MiB: 1024.00000 Copy: 2367.876 MiB/s
0 Method: MCBLOCK Elapsed: 0.13020 MiB: 1024.00000 Copy: 7865.125 MiB/s
AVG Method: MCBLOCK Elapsed: 0.13020 MiB: 1024.00000 Copy: 7865.125 MiB/s

and compared with dell r210 :
Long uses 8 bytes. Allocating 2*134217728 elements = 2147483648 bytes of memory.
Using 262144 bytes as blocks for memcpy block copy test.
Getting down to business... Doing 1 runs per test.
0 Method: MEMCPY Elapsed: 0.18472 MiB: 1024.00000 Copy: 5543.645 MiB/s
AVG Method: MEMCPY Elapsed: 0.18472 MiB: 1024.00000 Copy: 5543.645 MiB/s
0 Method: DUMB Elapsed: 0.12824 MiB: 1024.00000 Copy: 7985.277 MiB/s
AVG Method: DUMB Elapsed: 0.12824 MiB: 1024.00000 Copy: 7985.277 MiB/s
0 Method: MCBLOCK Elapsed: 0.06777 MiB: 1024.00000 Copy: 15110.377 MiB/s
AVG Method: MCBLOCK Elapsed: 0.06777 MiB: 1024.00000 Copy: 15110.377 MiB/s



If i compare cached reads from hdparm, i have the same problem :
Supermicro :
sudo hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 13684 MB in 2.00 seconds = 6850.07 MB/sec
Timing buffered disk reads: 1152 MB in 3.00 seconds = 383.49 MB/sec


Dell:
sudo hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 24470 MB in 2.00 seconds = 12250.99 MB/sec
Timing buffered disk reads: 368 MB in 3.01 seconds = 122.35 MB/sec


I designed this server for large scale database server but theses memory bandwidth problems give me poor cached hits performances.
On mysql benchmarks, 900€ dell r210 is twice better than 4000€ supermicro ...

I'm very disappointed...