LinuxQuestions.org - NFS Performance -- any ideas?

I have a pretty busy NFS server. There are a LOT of small I/O requests that come across. It is a cluster built on RHEL5 where I distribute each NFS export as a service across the cluster so it is in essence, an active/active cluster. Each export is ext3. Here is one of the mount points being shared out:
/dev/mapper/vg_data01a-lv_data01a on /data01a type ext3 (rw,noatime,nodiratime,data=writeback,commit=30)

I believe these to be optimum settings for performance, someone correct me if I am wrong.

I am starting 256 nfsd processes on each node. Here are the client mount settings:
omadvnfs01-nfs-h:/data01h /data01h nfs defaults,rsize=32768,wsize=32768 0 0

Here are some tests that drew me to NFS performance:

WRITING TO ISCSI DISK
[root@omadvnfs01b data01h]# time dd if=/dev/zero of=foo bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.547895 seconds, 490 MB/s

real 0m0.578s
user 0m0.004s
sys 0m0.540s
READING FROM ISCSI DISK
[root@omadvnfs01b data01h]# time dd if=foo of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 0.142407 seconds, 1.9 GB/s

real 0m0.144s
user 0m0.008s
sys 0m0.135s

WRITING TO NFS EXPORT
[root@omadvdss01a data01h]# time dd if=/dev/zero of=foo bs=16k count=16384
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 9.48628 seconds, 28.3 MB/s

real 0m14.394s
user 0m0.004s
sys 0m0.408s

READING FROM NFS EXPORT
[root@omadvdss01a data01h]# time dd if=foo of=/dev/null bs=16k
16384+0 records in
16384+0 records out
268435456 bytes (268 MB) copied, 4.03699 seconds, 66.5 MB/s

real 0m4.039s
user 0m0.007s
sys 0m0.302s

You can see the huge difference from direct I/O to the disk vs. nfs. I would like to close this gap a bit somehow. Any ideas?