Need tricks on tuning LVM and XEN IO subsystem.
 

Here's my setup:

2 TB of physical volumes in one volume group "Surplus". In there I have 3 Logical Volumes called OraASM_01, OraASM_02, OraASM_03. These are then mounted into Xen DomU running the Oracle ASM instance and DB. I'm noticing that inserts/deletes on two tables are going 50-75% slower than I think they should be.

I know hdparm -tT /dev/mapper/Surplus-OraASM_01 tests the read speed (which comes out at ~55MB/s while things are quiet) However while it's working iostat -m -x 2 is reporting 100% utilization at 25MB/s... but the drive is capable of 60MB/s?

The Oracle side I know how to look at, which I'm in the process of doing. It's the xen/LVM I'm falling a little short in knowledge.

1. How do I test the max read and write speeds of the LVM logical volume?
2. Is there a way to accurately see a LV's current working speed?
3. Is there a limit in the amount of IO instructions per second? If so how do I find what it's going and and what the max is?
4. Where could the bottleneck, assuming it's not on the Oracle side, and how do I detect it?