no, no, and no. (well.. maybe)

my setup -
64 bit PCI bus @ 133HMz
two dual-core xeons (so there appears to be 4 procs) - @ 800 MHz FSB
8 disk RAID array - 7200 rpm drives w/ 16MB disk buffers.
3ware 9550SX PCI-X RAID w/ 256MB memory - theoretically capable of 800MB/s (that's mega bytes, not mega bits) reads, 380 MB/sec writes

All that said, I don't think 1000Mb/sec is even possible - it's more like 750 Mb/sec once the packets are loaded and shipped. Regardless, it should be more than 250-300Mb/sec... Sure, maybe a SCSI array would be better, but at this point, hardware on the server is not the bottleneck.

BrianK

I ran accross and article entitled :'Squeeze Your Gigabit NIC for Top Performance". Not too much on Jumbo Frames but useful.(?) http://www.enterprisenetworkingplane...le.php/3485486

lazlow

Yes at this point there's no hardware problem, at the future surely it'll be affecting.

I never heard of one's getting fully GigE at 1g/sec(peak) but in 300Mbps you're right, it must be easy to achieve.

As comparison the D-core Xeon would be faster than D-core Pentium and withstand of dissipation.( I havent checked on Itanium for years)

We'll be happy to hear more from you........

Regards

link doesn't work.

BrianK

Oops!

http://www.enterprisenetworkingplane...le.php/3485486

lazlow

I just wondering on how we implement this setting on WiFi.

This thread is a little old but I am really surprised at the numbers people are discussing.

I can easily get ~600+ Mbps on a gigE PCI card (32bit 33 MHz) memory-memory.

With the onboard Nforce gigE adapters I can get 940 Mbps TCP throughput. Theoretical max is ~ 950 Mbps (if I accounted for the headers correctly). And this is between machines 1000km apart (but no routers in between).

Turning on jumbo frames (9000 bytes) I got 980 Mbps sustained on a link closer to 1500km as the crowflies (RTT 20ms.

For the long haul you need to increase the tcp window size but for normal LANs this will not make much if any difference.

disk-disk I have achieved > 800 Mbps throughput from Sydney to Perth (~3000 km) using softraid 0. This is all on ~ 2 GHz AMD (some dual CPU but that does not make much difference).

Regards
Chris

How are you testing this?
Do you have something abnormal in your configuration or did it work that way out of the box?
What distro?
What other hardware?

I'll admit I haven't tried computer-computer w/o a switch, but I've never seen anywhere near that speed. I'm curious (not doubting) how yours is so fast.

Testing using iperf (2.0.2) and my own custom written C program, which does memory-memory or disk-memory-memory-disk (or any of those sub-steps). Both iperf and my code gets the same numbers.

Debian Sarge.

Tyan Thunder K8WE mobo with dual AMD64 CPU (running 32bit kernel). This motherboard has dual ONBOARD gigE NICS which helps. But I get similar numbers from a Dell 1600SC with a PCI-X intel NIC and some Nforce3 "gaming" mobos

Note that this is not computer-computer. There are lots of switches in between (as I said about ~500-1000 km separated). There are no routes or firewalls (they are on layer-2 network).

This is all "out of the box" stuff - other than TCP window of 1-2 MBytes.

Next step is to get this rate from Sydney to Amsterdam... that may be more of a challenge. :-)

ttcp is useful for benchmarking. I consistantly see > 800 Mbit between two identically specced boxes using onboard nForce ethernet. Sending box was around 70% idle during test, according to 'top'. Receiving box appears to be maxed out though. Standard 1500 MTU (no jumbo frames).

Sending box:
Receiving box:
Hardware:
2 x AMD 3800+ on MSI K8N NEO2 Platinum-54G, 1GB PC3200 Corsair TWINX1024-3200XL, Mandriva 2006 x86_64, forcedeth driver
1 x CNet 8 port gigabit fanless switch

I would like to add some data to this, as I'm trying to get better throughput myself.

GigE is usually faster than a single HDD so testing file transfers is pointless. I'm testing with iperf.

http://dast.nlanr.net/Projects/Iperf/

It compiles easily in Slackware, there's a package for apt-get in Ubuntu, and it's available for Windows (good to test that environment too).

In summary, I think it all comes down to the quality of the hardware, and I have some interesting numbers.

At work I have two Intel servers; one server (DIME) is based on the Intel SE7500WV2 server board with Dual Xeon 2 Ghz processors. Both have dual on-board GigE Nics. The other (Terra) is an Intel SE7520JR2 serverboard with Dual Xeon 3.6 Ghz processors. Both are running Slackware 11.0 2.6.17.13, although with a recompiled Kernel to enable SMP, SMT and High Mem. Not much else has been tuned in the kernel.

Using iperf I can get about 939-940 Mbits/sec through the tcp test (one server runs iperf -s and the other runs iperf -c, no tuning of parameters) and I think that this is awesome. They are interconnected by a 3Com 3250 switch (3CR17501-91).

That's my best case scenario. No tuning of any networking parameter, but running quality hardware.

At home I have three machines with GigE, with vastly inferior hardware and significantly lower throughput. These machines are interconnected by a 3Com SuperStack 3 Switch 3812 (3C17401).

P360 is a Dell Precision 360 (purchased used), Gateway is a Celeron 400 based on a Gigabyte GA-6BXC and Overload is a Sempron 2800 on an ASUS K8N-E Deluxe. P360 and Overload have onboard GigE, and Gateway has had an Intel Pro/1000 GT Desktop adapter installed. All are regular PCI, 32-bit, 33 Mhz.

P360 is running Ubuntu Edgy and Gateway and Overload are running Slackware 11.0. Overload is on 2.6.17.13 and Gateway is on 2.6.18 (because it has a Hauppauge PVR-150 in it)

Now the numbers:
iperf to/from Gateway (from either other machine) I can only get about 250 Mbits/sec

iperf to P360, from Overload, I can get about 820 Mbits/sec, however from P360 to Overload I only get 690 Mbits/sec.

What I have noticed, is that the CPU load on Gateway and Overload is 100% during the test, and the other machines (the two servers and P360), barely get to 20%. BTW, I get the same numbers with a cross-over, but I know that my home switch doesn't support Jumbo frames, so this is all with a plain old 1500 MTU.

What's interesting is that all that CPU is in Interrupts. Running `vmstat 1` shows thousands of interrupts per second. This, I can't yet explain.

Here's some sample vmstat data, two lines per machine (not the first and not consecutive), before and during an iperf run, while running the server (iperf -s) The iperf client doesn't generate nearly as many interrupts.

The interrupt load on OVERLOAD is incredible, and I think GATEWAY simply can't handle it. (but that little box continues to surprise me - I'm hoping that I can find a "fix").

I must admit that this is the first thread I've read here on linuxquestions regarding GigE, and maybe this post is a bit premature (maybe the soln is in one of the other results) but I wanted to post my data in a seemingly relevant thread. I'll reply if I do find a relevant answer or solution.

._.

I'm still listening.......

I think I need to say that you all should change your cable to category 6 with a good AMP connector just in case to avoid semaphore.........dont you agree?

hmm as mentioned earlier, the max rate is a hardware bitrate but not a datastream, it neccessarily includes in it all the probably several packet wrappings and their attendant protocol's resends and crcs and whatnot.

your layer interfaces are wrapping and stripping as it passes in an out which of course is finite by hardware. if you send just one byte it gets wrapped into packets which could be wrapped in another or several headers, which is eating your bandwidth on the actual cable.

the double bus traversal of the data was also mentioned, then theres the disk write buffer and speed of the raid device. unless you increase the MTU size to epic proportion to decrease wrapper overhead you wont get anywhere near a gig on the cable, and what device barring perhaps a ramdisk can write at that speed anyway. but good luck with it.

edit: if you want to throw money at it get a couple of 2 nanosecond solid state ramdisks with a fibreoptic cable between them, that'd really fly.

ps. the large number of cpu interrupts above is, for my money, the cpu polling the bus to discover if its done doing something yet, that being transporting data.

Maybe this is out of the topic but I think still in the track.
I put my 3com940 GigBitEth peer to my iSCSI initiator(PCI-X 64 bit[not a PCI express]) then pull a 2 gigabit file from (sofware NAS) PC. File transfer is done in less than a minute.I compare this in 100 Mbps (Tell you what; I leave the PC and don't want to know).
My hardware is P4/3Gig and Xeon/1Gig of processor speed.
I think a full performance depending on how files were written in HDD+good interconnect+CPU&media bottleneck.
As I remembered that is between 300-400mpbs REAL MODE not peak.(I'm not sure)

Rubbish. Cat5e can saturate a gigE connection. Using jumbo frames (9000 MTU) I can get 989 Mbps user data throughputs with TCP. The 11 Mbps is ethernet/IP/TCP headers.


Chris