Linux - NetworkingThis forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
With 1000Mb/sec * 4 peak, don't you think you need Dual-Core processor/1600 Front Side Bus Mobo/15000rpm SCSI media?
no, no, and no. (well.. maybe)
my setup -
64 bit PCI bus @ 133HMz
two dual-core xeons (so there appears to be 4 procs) - @ 800 MHz FSB
8 disk RAID array - 7200 rpm drives w/ 16MB disk buffers.
3ware 9550SX PCI-X RAID w/ 256MB memory - theoretically capable of 800MB/s (that's mega bytes, not mega bits) reads, 380 MB/sec writes
All that said, I don't think 1000Mb/sec is even possible - it's more like 750 Mb/sec once the packets are loaded and shipped. Regardless, it should be more than 250-300Mb/sec... Sure, maybe a SCSI array would be better, but at this point, hardware on the server is not the bottleneck.
This thread is a little old but I am really surprised at the numbers people are discussing.
I can easily get ~600+ Mbps on a gigE PCI card (32bit 33 MHz) memory-memory.
With the onboard Nforce gigE adapters I can get 940 Mbps TCP throughput. Theoretical max is ~ 950 Mbps (if I accounted for the headers correctly). And this is between machines 1000km apart (but no routers in between).
Turning on jumbo frames (9000 bytes) I got 980 Mbps sustained on a link closer to 1500km as the crowflies (RTT 20ms.
For the long haul you need to increase the tcp window size but for normal LANs this will not make much if any difference.
disk-disk I have achieved > 800 Mbps throughput from Sydney to Perth (~3000 km) using softraid 0. This is all on ~ 2 GHz AMD (some dual CPU but that does not make much difference).
I can easily get ~600+ Mbps on a gigE PCI card (32bit 33 MHz) memory-memory.
With the onboard Nforce gigE adapters I can get 940 Mbps TCP throughput. Theoretical max is ~ 950 Mbps (if I accounted for the headers correctly). And this is between machines 1000km apart (but no routers in between).
Turning on jumbo frames (9000 bytes) I got 980 Mbps sustained on a link closer to 1500km as the crowflies (RTT 20ms.
How are you testing this?
Do you have something abnormal in your configuration or did it work that way out of the box?
What distro?
What other hardware?
I'll admit I haven't tried computer-computer w/o a switch, but I've never seen anywhere near that speed. I'm curious (not doubting) how yours is so fast.
How are you testing this?
Do you have something abnormal in your configuration or did it work that way out of the box?
What distro?
What other hardware?
I'll admit I haven't tried computer-computer w/o a switch, but I've never seen anywhere near that speed. I'm curious (not doubting) how yours is so fast.
Testing using iperf (2.0.2) and my own custom written C program, which does memory-memory or disk-memory-memory-disk (or any of those sub-steps). Both iperf and my code gets the same numbers.
Debian Sarge.
Tyan Thunder K8WE mobo with dual AMD64 CPU (running 32bit kernel). This motherboard has dual ONBOARD gigE NICS which helps. But I get similar numbers from a Dell 1600SC with a PCI-X intel NIC and some Nforce3 "gaming" mobos
Note that this is not computer-computer. There are lots of switches in between (as I said about ~500-1000 km separated). There are no routes or firewalls (they are on layer-2 network).
This is all "out of the box" stuff - other than TCP window of 1-2 MBytes.
Next step is to get this rate from Sydney to Amsterdam... that may be more of a challenge. :-)
ttcp is useful for benchmarking. I consistantly see > 800 Mbit between two identically specced boxes using onboard nForce ethernet. Sending box was around 70% idle during test, according to 'top'. Receiving box appears to be maxed out though. Standard 1500 MTU (no jumbo frames).
It compiles easily in Slackware, there's a package for apt-get in Ubuntu, and it's available for Windows (good to test that environment too).
In summary, I think it all comes down to the quality of the hardware, and I have some interesting numbers.
At work I have two Intel servers; one server (DIME) is based on the Intel SE7500WV2 server board with Dual Xeon 2 Ghz processors. Both have dual on-board GigE Nics. The other (Terra) is an Intel SE7520JR2 serverboard with Dual Xeon 3.6 Ghz processors. Both are running Slackware 11.0 2.6.17.13, although with a recompiled Kernel to enable SMP, SMT and High Mem. Not much else has been tuned in the kernel.
Using iperf I can get about 939-940 Mbits/sec through the tcp test (one server runs iperf -s and the other runs iperf -c, no tuning of parameters) and I think that this is awesome. They are interconnected by a 3Com 3250 switch (3CR17501-91).
That's my best case scenario. No tuning of any networking parameter, but running quality hardware.
At home I have three machines with GigE, with vastly inferior hardware and significantly lower throughput. These machines are interconnected by a 3Com SuperStack 3 Switch 3812 (3C17401).
P360 is a Dell Precision 360 (purchased used), Gateway is a Celeron 400 based on a Gigabyte GA-6BXC and Overload is a Sempron 2800 on an ASUS K8N-E Deluxe. P360 and Overload have onboard GigE, and Gateway has had an Intel Pro/1000 GT Desktop adapter installed. All are regular PCI, 32-bit, 33 Mhz.
P360 is running Ubuntu Edgy and Gateway and Overload are running Slackware 11.0. Overload is on 2.6.17.13 and Gateway is on 2.6.18 (because it has a Hauppauge PVR-150 in it)
Now the numbers:
iperf to/from Gateway (from either other machine) I can only get about 250 Mbits/sec
iperf to P360, from Overload, I can get about 820 Mbits/sec, however from P360 to Overload I only get 690 Mbits/sec.
What I have noticed, is that the CPU load on Gateway and Overload is 100% during the test, and the other machines (the two servers and P360), barely get to 20%. BTW, I get the same numbers with a cross-over, but I know that my home switch doesn't support Jumbo frames, so this is all with a plain old 1500 MTU.
What's interesting is that all that CPU is in Interrupts. Running `vmstat 1` shows thousands of interrupts per second. This, I can't yet explain.
Here's some sample vmstat data, two lines per machine (not the first and not consecutive), before and during an iperf run, while running the server (iperf -s) The iperf client doesn't generate nearly as many interrupts.
Code:
DIME:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 2528 2119216 67596 894876 0 0 0 0 255 255 0 0 100 0
0 0 2528 2118704 67596 894876 0 0 0 0 8288 16760 0 14 85 0
TERRA: (was busy, but in jumps up and id is not 0)
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 41100 217552 60516 3550432 0 0 51 583 329 11313 41 7 50 2
4 0 41100 213844 60664 3551032 0 0 32 382 6808 19038 38 22 39 1
P360:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
0 0 0 1362548 60496 430396 0 0 0 3 350 258 3 1 97 0
1 0 0 1361516 60496 430396 0 0 0 6 8281 16180 3 24 73 0
OVERLOAD:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
0 0 2644 14824 150488 666396 0 0 0 0 450 97 0 1 99 0
2 0 2644 12344 150488 666396 0 0 0 0 32796 548 0 100 0 0
GATEWAY:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
1 0 2568 255308 23164 31644 0 0 0 9 254 212 0 0 100 0
2 0 2568 255052 23224 31644 0 0 0 9 4918 617 1 99 0 0
The interrupt load on OVERLOAD is incredible, and I think GATEWAY simply can't handle it. (but that little box continues to surprise me - I'm hoping that I can find a "fix").
I must admit that this is the first thread I've read here on linuxquestions regarding GigE, and maybe this post is a bit premature (maybe the soln is in one of the other results) but I wanted to post my data in a seemingly relevant thread. I'll reply if I do find a relevant answer or solution.
._.
Last edited by Slim Backwater; 03-10-2007 at 05:34 AM.
I think I need to say that you all should change your cable to category 6 with a good AMP connector just in case to avoid semaphore.........dont you agree?
hmm as mentioned earlier, the max rate is a hardware bitrate but not a datastream, it neccessarily includes in it all the probably several packet wrappings and their attendant protocol's resends and crcs and whatnot.
your layer interfaces are wrapping and stripping as it passes in an out which of course is finite by hardware. if you send just one byte it gets wrapped into packets which could be wrapped in another or several headers, which is eating your bandwidth on the actual cable.
the double bus traversal of the data was also mentioned, then theres the disk write buffer and speed of the raid device. unless you increase the MTU size to epic proportion to decrease wrapper overhead you wont get anywhere near a gig on the cable, and what device barring perhaps a ramdisk can write at that speed anyway. but good luck with it.
edit: if you want to throw money at it get a couple of 2 nanosecond solid state ramdisks with a fibreoptic cable between them, that'd really fly.
ps. the large number of cpu interrupts above is, for my money, the cpu polling the bus to discover if its done doing something yet, that being transporting data.
Maybe this is out of the topic but I think still in the track.
I put my 3com940 GigBitEth peer to my iSCSI initiator(PCI-X 64 bit[not a PCI express]) then pull a 2 gigabit file from (sofware NAS) PC. File transfer is done in less than a minute.I compare this in 100 Mbps (Tell you what; I leave the PC and don't want to know).
My hardware is P4/3Gig and Xeon/1Gig of processor speed.
I think a full performance depending on how files were written in HDD+good interconnect+CPU&media bottleneck.
As I remembered that is between 300-400mpbs REAL MODE not peak.(I'm not sure)
I think I need to say that you all should change your cable to category 6 with a good AMP connector just in case to avoid semaphore.........dont you agree?
Rubbish. Cat5e can saturate a gigE connection. Using jumbo frames (9000 MTU) I can get 989 Mbps user data throughputs with TCP. The 11 Mbps is ethernet/IP/TCP headers.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.