I have a DB server setup I use for application/product performance testing.

The DB server RHEL environment is currently configured as follows:
- IBM x3650 M4 server
- RHEL 6.7 running on vmware ESXI 5.5
- internal disks + internal SSDs

Usually when configuring performance tests for the DB servers, I would configure the data files on local disk and the DB transaction logs (the usual bottleneck) on local SSD to see how far the application can be pushed.

The current application/product I'm testing has the DB server storage configured with LUKs.

While running performance load, I couldn't drive the SSD to anywhere near its limits.
At the same time, when watching mpstat I could see lots of IOWAIT time.
After a lot of going round in circles I discovered:

- when I moved the DB transaction logs to an unencrypted filesystem, on the same SSD, the IO/sec rate to the SSD doubled (and the IOWAIT time became negligible - all other storage areas were still configured with LUKS);

- when I moved the DB transaction logs back to a new LUKS configured filesystem on the same SSD, the SSD throughput/high IOWAIT problem came back.

LUKS proved to be the root of the performance issue.
Hence my question is:
- is there any config I can do to alleviate this issue/better configure my RHEL environment/kernel parameters for LUKS or do I just accept that if I use LUKS in a high-throughput environment and LUKS encryption/decryption can't keep up with storage throughput capacity, then such is life ?

[For info, I've done a lot of searching on this topic but I haven't found anything recent re LUKS performance.]

Do you have AES support in the processor? What does "grep aes /proc/cpuinfo" report? Intel's spec sheet just says, "Some products can support AES New Instructions with a Processor Configuration update," and encryption is going to be slow without that.

Thank you rknichols for taking an interest in this.
The response from the grep aes /proc/cpuinfo is as follows:
(output was repeated for each cpu)

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc
arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinl
ock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_
lm ida arat epb pln pts dts

Probably a matter of "suck it up".
The database is no doubt designed to expect direct I/O, and you've slapped a LUKS container in the way that intercepts every I/O and then [de-]mangles every bit that goes in either direction. Yeah, I know, it wasn't your choice - ain't life a bitch.

Maybe look to make sure you have passthrough defined for that guest, and "cryptsetup benchmark", but you may be just tinkering around the edges. If you can't get the I/O done in a timely manner, you gotta wait.

Thanks for the reply syg00. I did try configuring the server in passthrough. Vms wouldn't start afterwards so I had to undo it. Still looking at various doc to see if there's another way but I think it's leading to having to "suck it up" or else find an alternative solution.