How to reach near-Gb/s speed on my network?

Brol · 08-25-2009, 11:50 AM

Hi,

Do you have any hints on how I could tune my network and get near Gb/s speed?

Using ttcp, I'm getting only about 620 Mb/s.

[root@source_gb ~]# ttcp -t -s -p 2000 -l 1000000 target_gb
ttcp-t: buflen=1000000, nbuf=2048, align=16384/0, port=2000 tcp -> barbados_priv
ttcp-t: socket
ttcp-t: connect
ttcp-t: 2048000000 bytes in 25.44 real seconds = 78630.84 KB/sec +++
ttcp-t: 2048 I/O calls, msec/call = 12.72, calls/sec = 80.52
ttcp-t: 0.0user 8.6sys 0:25real 34% 0i+0d 0maxrss 0+245pf 4533+1171csw

Thanks for your help!

orgcandman · 08-25-2009, 12:05 PM

The question you have asked is entirely too vague to be able to give an accurate answer.

A "fast" network is dependent on a lot of things, some of which are simply "which protocol have you chosen?" However, many questions involve what hardware is being run, what software is being run, what kind of cabling is used, and how long is it, etc..

lazlow · 08-25-2009, 01:08 PM

Remember to actually use gigabit speeds you have to have the hard drives on both ends to be able to handle it. 1Gb/s is about 95MB/s after overhead. While some newer(consumer) drives can handle this speed (and most raid0 setups) a lot of older drives cannot. The 620Mb/s would be about 77MB/s(before overhead), which would be about where a lot of drives from just a few years ago would fall. You can check how fast your drives are using hdparm (see man hdparm).

Quote:

[root@localhost ~]# hdparm -tT /dev/sda

/dev/sda:
Timing cached reads: 2732 MB in 2.00 seconds = 1366.11 MB/sec
Timing buffered disk reads: 226 MB in 3.01 seconds = 75.15 MB/sec

from a 320GB(above) or on an little older 250gb drive(below):

Quote:

root@localhost ~]# hdparm -tT /dev/hda

/dev/hda:
Timing cached reads: 2568 MB in 2.00 seconds = 1284.45 MB/sec
Timing buffered disk reads: 174 MB in 3.03 seconds = 57.41 MB/sec

So a transfer from one machine to another (using those two drives) would be limited to 57MB/sec.

Brol · 08-25-2009, 01:58 PM

Ok, let me add some more details.

Hardware:

Server1:
Motherboard: Asus P5E-V HDMI
CPU: Intel Core2 6600 2.4Ghz
FSB: 1066Mhz
RAM: 4GB (800Mhz)
NIC: PCI D-Link DGE-528T full duplex gigabit

Server2:
Motherboard: Intel S5000PAL
CPU: 2x Intel Xeon Quad E5430 2.66Ghz
FSB: 1333Mhz
RAM: 2GB (667Mhz)
NIC: 2x integrated gigabit

Switch:
D-Link DGS-1216T Smart gigabit switch

Cables:
CAT-6 1m

Software:

OS:
Both servers are running Fedora FC 11 x64.

Protocol:
TCP

The network is private, i.e. only these 2 servers are on this network. I have left all network-related parameters to their default values (e.g. MTU=1500).

Brol · 08-25-2009, 02:16 PM

Quote:

Originally Posted by lazlow

Remember to actually use gigabit speeds you have to have the hard drives on both ends to be able to handle it. 1Gb/s is about 95MB/s after overhead. While some newer(consumer) drives can handle this speed (and most raid0 setups) a lot of older drives cannot. The 620Mb/s would be about 77MB/s(before overhead), which would be about where a lot of drives from just a few years ago would fall.

Yes, I thought about that, that's why I used ttcp which (presumably) does a memory-to-memory test, bypassing any disk I/O. However, it did strike me when I noticed that ttcp reports approximately the same performance as hdparm does:

[root@target_gb ~]# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 228 MB in 3.00 seconds = 76.00 MB/sec

ttcp reports: 78630.84 KB/sec

Could it be just a coincidence?..

Anyway, I made another test. On my source server, I have a RAID-5 Volume which is much faster in read:

[rootsource_gb~]# hdparm -t /dev/sdb1

/dev/sdb1:
Timing buffered disk reads: 950 MB in 3.00 seconds = 316.39 MB/sec

I made the following test to read from the RAID, transfer to the network, and dump to /dev/null on the target:

[root@source_gb mnt]# dd if=/mnt/1g ibs=1M | ( ssh target_gb dd of=/dev/null obs=1M)
1024+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 29.3306 s, 36.6 MB/s
2097152+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 29.0326 s, 37.0 MB/s

This way I only get about 40 MB/s, which is what I have also observed with scp. With ftp I could reach about 65 MB/s. Which is why I decided to bypass any disk I/O and use ttcp. The target_gb server is anyway supposed to be diskless.

lazlow · 08-25-2009, 03:41 PM

The buffered read really do not tell you much. It is when you go beyond the buffer(-T) that it becomes important. However your buffered read seems extremely slow. Even on my older disks(5? years old) my buffered number is 1284.45 MB/sec compared to your 316.39 MB/sec. What kind of raid card are you using?

Brol · 08-25-2009, 05:03 PM

Quote:

Originally Posted by lazlow

The buffered read really do not tell you much. It is when you go beyond the buffer(-T) that it becomes important. However your buffered read seems extremely slow. Even on my older disks(5? years old) my buffered number is 1284.45 MB/sec compared to your 316.39 MB/sec. What kind of raid card are you using?

You mean, your CACHED reads, right? If I add the -T option, I get:

[root@source_gb mnt]# hdparm -tT /dev/sdb1

/dev/sdb1:
Timing cached reads: 7980 MB in 2.00 seconds = 3995.88 MB/sec
Timing buffered disk reads: 954 MB in 3.00 seconds = 317.63 MB/sec

But anyway, back to the original point: when using ttcp I'm bypassing the disk I/O, and this is exactly what I want to do, ie. measure and tune the network only. I'm limited to 620 Mb/s, and I'd like to reach near-Gb/s performance.

lazlow · 08-25-2009, 06:17 PM

Oops, guess I need more caffeine.

Have you tried bypassing the switch to see if it is the issue? When using GigE you do NOT need a crossover cable.