Ive been looking for a tool to perform LUN I/O performance testing.
I have N number of luns.
On each lun I want to start multiple parallel jobs/threads performing I/O.
The I/O should be large block, 64/128/256k and completely random, with little if any sequential.

I looked at a tool called fio, but im not sure exactly how to configure an example which meets the above conditions. The do have a "--bs" option, which I can use and I found "rw=randrw" will do the random, but I don't know how to define the number of luns and number of threads per lun.

Any clarification would be appreciated, or if there is another tool that would be helpful as well.

Do you want to benchmark filesystem access times or do you want raw disk speed? fio does file I/O. Another good filesystem benchmark is Bonnie++.

If you want to test the block device without going through the filesytem and cache, then use iometer or vdbench.

What I have is a handfull of raw device luns, to be used for a block-oriented database. These luns could be fiber or iscsi, but in any event are backed by a fairly robust storage array.
I tried fio, and got rediculous numbers. It was telling me that I was getting about 5-10 MB/sec from a fiber mounted lun striped accross a Clariion disk array with hundreds of drives and little other workload.
I admit I don't know what the options are trying to tell me, so perhaps im using it wrong. When I tried 'dd', I got upwards of 1300 MByte/sec (1.3 GByte/sec) throughput, but fio was giving me less than 10 MByte/sec. Something not right with that program (or how im using it).
Best I could tell from reading other sites, the following options should have worked for me.
./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/pool/test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75

A good database layout will spread I/O equally across all devices and the database will use cache to minimize slow seeks. If you can find a copy of Orion it simulates database activity.

Try running fio without --direct=1. Using direct I/O will bypass disk and kernel buffer caches and measure only the underlying hardware performance. This likely accounts for some of the performance loss between dd and fio. Also, dd will do sequential I/O, but you've asked fio to perform a slight read heavy random I/O pattern. So you're really comparing apples to oranges here.

Actually, in my experience, the most reliable way to predict these things is through software simulation. Specifically, "discrete event simulation." There are many tools out there in open-source land which can help you do this, "sufficiently well to be meaningful."

I'm not 100% sure from your description exactly how the database system that you wish to model is set up; that is to say, what sort of request pattern it would generate and how an operating system might (if tuned in various ways) respond to it. But, that will be the most important aspect of your simulation. If you anticipate this correctly, your simulation should very quickly and correctly identify the "hot spots." (And, that's really all that you need to know ...) If you don't, the results will be garbage.