Hi, i'm building a Debian8 x64 box and hoped to get some configuration advice from everyone here. Here is my hardware -

2x Xeon E5-2620 v4 @ 2.10Ghz
128GB Memory
Supermicro X10DAI
2x LSI 9361-8i PCIe 3.0 Controllers (1 full and 1 empty)
8x 1TB Samsung 850 Pro SSD (Raid5 on one of the controllers, no hot spare)
2x 512GB Samsung 950 Pro NVMe (PCIe 3.0)

The server is running VMWare ESXi 6.0 to separate the hardware from the guest OS allowing easier upgrades, however it will be a dedicated host for this guest Debian server. The Debian server will be running ElasticSearch for indexing where Read Performance is the priority rather than Write.

Inside my Debian VM I have added a drive from the Raid5 volume and 2 drives that map to each of the NVMe cards. My initial thought was to create a standard pool with the Raid5 volume and then add the two NVMe cards as cache devices, but wanted to know if anyone else has any suggestions around getting every last bit of performance out of this?

zpool create -f pool0 /dev/sdb -m /zfs
zpool add pool0 -f cache /dev/sdc /dev/sdd
zfs set sync=disabled pool0

For benchmarking I spun up a Windows 2012R2 VM and ran Crystal Disk Mark against the volumes that way I had a more familiar baseline, in Debian I was using Bonnie++

Config 1): 2x NVMe (Software Raid0) - Windows 2012 R2

Seq Q32T1 - Read: 3219MB/s - Write: 3065MB/s
4K Q32T1 - Read: 447MB/s - Write: 436MB/s
Seq - Read: 2574MB/s - Write: 2511MB/s
4K - Read: 41MB/s - Write: 96MB/s

Config 2): 8x Samsung SSD (Raid5) - Windows 2012 R2

Seq Q32T1 - Read: 4137MB/s - Write: 2932MB/s
4K Q32T1 - Read: 493MB/s - Write: 156MB/s
Seq - Read: 2544MB/s - Write: 2357MB/s
4K - Read: 30MB/s - Write: 65MB/s

Config 3): 2x NVMe (mirror) running ZFS6

Output Block: 755M/sec
Output Rewrite: 551M/s
Input Block: 1494M/s

Config 4): 8x SSD in Raid5, 2x NVMe (cache) running ZFS6

Output Block: 295M/sec
Output Rewrite: 168M/s
Input Block: 1539M/s

Config 5): 8x SSD in Raid5, 2x NVMe (cache) running ZFS6 - ashift=12 + deadline scheduler

Output Block: 300M/sec
Output Rewrite: 171M/s
Input Block: 1522M/s

Config 6): 8x SSD in Raid5, 2x NVMe (cache) running bcache

Output Block: 534M/sec
Output Rewrite: 236M/s
Input Block: 1171M/s

I know i'm not really comparing apples and apples with the Crystal test but I thought they should at least be similar? Does anyone have any suggestions on how to improve this?

Any guidance much appreciated!
Thanks

What's the question? You're asking about windows vs linux software? Or the report format? MB/s vs. M/s?

Hi,

I was really asking if anyone could suggest a better configuration that would perform the same or similar to the Windows based results.

Thanks

Providing a side-by-side comparison of the results that are easy to read might make it easier for you to get feedback.