Would you be more specific and give details?
Like hardware spec/config/net distance etc......
That will prove your state.
How about IpV6 have you try that?

??? I have nothing to "prove". I routinly transfer data between sites 500km on a dedicated 1 Gbps network. Any decent (intel pro/1000, nvidia etc) NICs can easily satutate the network.

The tricky bit is transfering data internationally Australia to Europe has a 330msec RTT. That makes TCP transfers difficult. But I can get > 700 Mbps with UDP.

Never tried IPv6 no need yet.

The bottom line is any modern mobo and a half decent NIC (not realtek they are crap!) can sustain 1 Gbos.

Because I'm still having a problem duplicating your results, I hookup up two beefy file servers with GigE Intel NICs using a crossover cable - computer to computer, no switches, maybe 3 meters of cable, MTU of 9000 (actually 9014) on both machines. I still only get 3-400Mb/s.

If you would be so kind, could you post your timing data? i.e. the result of "time dd if=/dev/zero of=/nfs/share/foo bs=1M count=1000" - that's assuming, of course, you're using NFS. If not, what are you using for transfers? could you post some test results showing the speeds you're getting?

Thanks.

Oh - your complaining that you cannot get decent performance on nfs over gigabit links. Well that is another question entirely!

I am taking about ethernet TCP speeds - this is memory to memory. If you are using nfs then the speeds could be terrible. Have you tested the raw speed of the device you are writing to? A single disk could max out at 300-400 Mbps plus the substantial nfs overhead.

I use all my own software for network (and disk timing). Network transfer of data is my job to I cannot reply on tools such as "dd".

Using my own software which reads from a file, sends down a TCP socket and does the opposite on a second PC I can sustain about 600 Mbps (disk-to-disk). This is limited by the RAID array write speed (remember write speeds are usually a lot slower than read speed).

CPU usage is < 20%. This is a dual opteron (242?) system with a Tyan mobo. It is 2 years old.

You may want to check stuff like TCP offload. The tcp segmetation offload on the pci express intel pro/1000 desktop seems totally broken.

Cheers
Chris

Ah ha! Now we're on to something. It sounds like there is not an available protocol for sharing files at such high speeds - nfs, samba, ftp, etc. etc. etc. actually do have some sort of limit of around 3-400 Mb/s (what everyone here has been seeing). Does that sound accurate? Is your code open source? If so, where is it, I'd love to check it out.

For the record, yes, I've tested write speed on the receiving device to be around 70MB/s (that's bytes not bits). I've also tested using nfs, ftp, smb, and scp (and whatever windows-windows computers use for transfers) - all with pretty similar results, in several different facilities.

FTP has no such limit - here's 88 MB/s over GigE. Since the server processor was maxed out during the transfer, I'd expect more if my server had a faster processor or less interrupt-hungry NIC or was configured for jumbo frames. The key to speed here is that the file was served from a ramdisk and saved to /dev/null. The server and client hard drives can handle only about half that speed and would ~half the transfer rate if used as a source or destination.

# wget -O /dev/null --ftp-user={REDACTED} --ftp-password={REDACTED} ftp://192.168.1.50/ramdisktest
--16:57:50-- ftp://192.168.1.50/ramdisktest
=> `/dev/null'
Connecting to 192.168.1.50:21... connected.
Logging in as {REDACTED} ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD not needed.
==> PASV ... done. ==> RETR ramdisktest ... done.
Length: 512,000,000 (488M) (unauthoritative)

100%[===========================================>] 512,000,000 88.81M/s ETA 00:00

16:57:56 (88.14 MB/s) - `/dev/null' saved [512000000]


Furthermore, here is the same test done locally on the server showing just shy of 200 MB/s over FTP via the loopback interface:
# wget -O /dev/null --ftp-user={REDACTED} --ftp-password={REDACTED} ftp://192.168.1.50/ramdisktest
--17:28:40-- ftp://192.168.1.50/ramdisktest
=> `/dev/null'
Connecting to 192.168.1.50:21... connected.
Logging in as {REDACTED} ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD not needed.
==> PASV ... done. ==> RETR ramdisktest ... done.
Length: 512,000,000 (488M) (unauthoritative)

100%[===========================================>] 512,000,000 198.34M/s

17:28:42 (198.29 MB/s) - `/dev/null' saved [512000000]

I've had a brief look at serving from the ramdisk over GigE with nfs, smb, and scp too...

nfs: Unfortunately tmpfs (used to create my ramdisk) is one of the local filesystems known not to work with Linux NFS servers. The tmpfs filesystem was not visible over nfs - the directory it was mounted on showed instead.

smb: Much slower - best I've seen was about 9.5 MB/s. Not CPU-bound either as server and client both showed ~90% idle during transfer. This was surprising. Maybe there are samba tweaks to speed this up.

scp: Also slow at 13.6MB/s but server processor was maxed out with ssh consuming 99% so I'd expect more with a faster processor.

Actually I would like to second the thanks for information, this has been a question of mine as well, I have a setup similar to the OP's and this info has helped me figure out a few new things. Muchos Gracias!