Slow download fast upload to file server
 

Hi

I have strange behaviour of my file server. If I send 10 files of 100MB to it, it always takes about 35seconds. But if I download them again (to the same location where it came from) the time is random from 35sec up to 5min!

This is what I use as server:
samba 3.0.12 on suse9.3
cpu:sempron 2400+ /chipset:nForce2 /ram:250MB ddr400 /hd:40GB WD /DVDRW228 Philips /lan: onboard gigabit

For storage of files I created a software raid 0 with 2x 120GB IDE Hitachi drives

Benchmark of storage:
xxxx:/storage1 # bonnie -s 1000
Bonnie 1.4: File './Bonnie.13458', size: 1048576000, volumes: 1
Writing with putc()... done: 27817 kB/s 93.9 %CPU
Rewriting... done: 20689 kB/s 14.7 %CPU
Writing intelligently... done: 59551 kB/s 26.9 %CPU
Reading with getc()... done: 25651 kB/s 87.0 %CPU
Reading intelligently... done: 50612 kB/s 20.7 %CPU
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
xxxx 1*1000 27817 93.9 59551 26.9 20689 14.7 25651 87.0 50612 20.7 183.5 1.8

This is the client:
winXP
cpu:3Ghz p4 /ram:1GB /hd:37GB WD raptor /lan: onboard gigabit

Back to the server:
I checked DMA settings from the hard disks and they are switched on.
When I do the "corrupt" download the CPU load on the fileserver is normal; about 15%.
I checked up and downloading with FTP and this gives a similar result. Up is good but downloading is bad. The behaviour with FTP is even worse than with samba. FTP downloads in burst with long periods of radio silence (20 seconds or more) after each "burst" of data the file that was interrupted is left incomplete and corrupt!

Ethereal dump of FTP download (x.x.x.254=server): after the [segment lost] the download stalls...
No. Time Source Destination Protocol Info
13953 10.185560 192.168.0.254 192.168.0.10 FTP-DATA FTP Data: 1460 bytes
13954 10.185586 192.168.0.254 192.168.0.10 FTP-DATA FTP Data: 1460 bytes
13955 10.185621 192.168.0.10 192.168.0.254 TCP enpc > 5085 [ACK] Seq=1 Ack=38928385 Win=11711 Len=0
13956 10.185721 192.168.0.10 192.168.0.254 TCP [TCP Window Update] enpc > 5085 [ACK] Seq=1 Ack=38928385 Win=19631 Len=0
13957 10.185749 192.168.0.254 192.168.0.10 FTP-DATA FTP Data: 1460 bytes
13958 10.185760 192.168.0.254 192.168.0.10 FTP-DATA FTP Data: 1460 bytes
13959 10.387581 192.168.0.254 192.168.0.10 FTP-DATA [TCP Previous segment lost] FTP Data: 1295 bytes
13960 10.574757 192.168.0.10 192.168.0.254 TCP [TCP ZeroWindow] enpc > 5085 [ACK] Seq=1 Ack=38993920 Win=0 Len=0
13961 10.746771 192.168.0.10 192.168.0.255 NBNS Name query NB ZONELABS.COM<00>
13962 10.799515 192.168.0.254 192.168.0.10 TCP [TCP Keep-Alive] 5085 > enpc [ACK] Seq=38993919 Ack=1 Win=5840 Len=0
13963 10.799673 192.168.0.10 192.168.0.254 TCP [TCP ZeroWindow] [TCP Keep-Alive ACK] enpc > 5085 [ACK] Seq=1 Ack=38993920 Win=0 Len=0
13964 11.249447 192.168.0.254 192.168.0.10 TCP [TCP Keep-Alive] 5085 > enpc [ACK] Seq=38993919 Ack=1 Win=5840 Len=0
13965 11.249602 192.168.0.10 192.168.0.254 TCP [TCP ZeroWindow] [TCP Keep-Alive ACK] enpc > 5085 [ACK] Seq=1 Ack=38993920 Win=0 Len=0
13966 11.496762 192.168.0.10 192.168.0.255 NBNS Name query NB ZONELABS.COM<00>
13967 12.149312 192.168.0.254 192.168.0.10 TCP [TCP Keep-Alive] 5085 > enpc [ACK] Seq=38993919 Ack=1 Win=5840 Len=0
13968 12.149465 192.168.0.10 192.168.0.254 TCP [TCP ZeroWindow] [TCP Keep-Alive ACK] enpc > 5085 [ACK] Seq=1 Ack=38993920 Win=0 Len=0
13969 12.166434 192.168.0.254 192.168.0.10 SSH Encrypted response packet len=1460
13970 12.166455 192.168.0.254 192.168.0.10 SSH Encrypted response packet len=1376
13971 12.166749 192.168.0.10 192.168.0.254 TCP bvcontrol > ssh [ACK] Seq=208 Ack=11412 Win=62699 Len=0
13972 12.166775 192.168.0.10 192.168.0.254 TCP [TCP Window Update] bvcontrol > ssh [ACK] Seq=208 Ack=11412 Win=65535 Len=0


With samba I never experienced that files were not downloaded completely, but downloading is too slow. I did the FTP test to rule out samba problems.
If I look at an Ethereal dump of an upload and compare it to an download (both with same file with samba). I see different amount of packages. upload:47294 / download:69305 In my opinion this indicates less efficiency during download but I do not know why. I rule out hardware problems because the upload is ok.

This is where my knowledge ends. I am searching the www for days now to find a answer without success. I hope someone can give me a hint.

Thanx in advance.

Have you tried running 'netpipe' to check the efficacy of your network connection?

Euh netpipe? Sorry but I am no expert on TCP-IP, but I will try it. Can you explain what to look for?

I really hope it will gives some answers.

I did another test today... I made my winXP dual boot and put suse 9.2 on it to test the connection linux to linux.

Result of copy 11 files togeter 1GB:
upload to server : 45sec
download from server: 45sec

I did it several times and it is stable! (somehow the upload is a little bit slower than when I use winXP to upload) My problem seems to be on the winXP side, but what can I do about it?

Maybe I will also try to install winXP on the server just to see… But I rather don’t.

I go try to install netpipe now.

Hi,

I run NetPIPE a few times form server side and client side. All results are almost the same. Please tell me what I learn from this.:scratch:

How can these numbers help me solve the slow downloads to winXP?
(NetPIPE works only linux <-> linux)

Here is the result:
xxxx:/usr/src/NetPIPE_3.6.2/bin # ./NPtcp -h 192.168.0.10
Send and receive buffers are 16384 and 87380 bytes
(A bug in Linux doubles the requested buffer sizes)
Now starting the main loop
0: 1 bytes 1370 times --> 0.12 Mbps in 62.51 usec
1: 2 bytes 1599 times --> 0.06 Mbps in 246.15 usec
2: 3 bytes 406 times --> 0.09 Mbps in 249.97 usec
3: 4 bytes 266 times --> 0.12 Mbps in 249.96 usec
4: 6 bytes 300 times --> 0.18 Mbps in 249.96 usec
5: 8 bytes 200 times --> 0.24 Mbps in 249.94 usec
6: 12 bytes 250 times --> 0.37 Mbps in 249.96 usec
7: 13 bytes 166 times --> 0.40 Mbps in 249.95 usec
8: 16 bytes 184 times --> 0.49 Mbps in 249.94 usec
9: 19 bytes 225 times --> 0.58 Mbps in 249.96 usec
10: 21 bytes 252 times --> 0.64 Mbps in 249.95 usec
11: 24 bytes 266 times --> 0.73 Mbps in 249.96 usec
12: 27 bytes 283 times --> 0.82 Mbps in 249.96 usec
13: 29 bytes 177 times --> 0.89 Mbps in 249.93 usec
14: 32 bytes 193 times --> 0.98 Mbps in 249.94 usec
15: 35 bytes 212 times --> 1.07 Mbps in 249.95 usec
16: 45 bytes 228 times --> 1.37 Mbps in 249.96 usec
17: 48 bytes 266 times --> 1.47 Mbps in 249.96 usec
18: 51 bytes 275 times --> 1.56 Mbps in 249.95 usec
19: 61 bytes 156 times --> 1.86 Mbps in 249.93 usec
20: 64 bytes 196 times --> 1.95 Mbps in 249.95 usec
21: 67 bytes 206 times --> 2.05 Mbps in 249.95 usec
22: 93 bytes 214 times --> 2.84 Mbps in 249.95 usec
23: 96 bytes 266 times --> 2.93 Mbps in 249.96 usec
24: 99 bytes 270 times --> 3.02 Mbps in 249.96 usec
25: 125 bytes 145 times --> 3.82 Mbps in 249.87 usec
26: 128 bytes 198 times --> 3.91 Mbps in 249.94 usec
27: 131 bytes 203 times --> 4.00 Mbps in 249.95 usec
28: 189 bytes 207 times --> 5.79 Mbps in 249.10 usec
29: 192 bytes 267 times --> 5.86 Mbps in 249.96 usec
30: 195 bytes 268 times --> 5.95 Mbps in 249.96 usec
31: 253 bytes 139 times --> 7.72 Mbps in 249.92 usec
32: 256 bytes 199 times --> 7.81 Mbps in 249.93 usec
33: 259 bytes 201 times --> 7.91 Mbps in 249.94 usec
34: 381 bytes 203 times --> 11.63 Mbps in 249.95 usec
35: 384 bytes 266 times --> 11.72 Mbps in 249.96 usec
36: 387 bytes 267 times --> 11.81 Mbps in 249.94 usec
37: 509 bytes 136 times --> 15.54 Mbps in 249.93 usec
38: 512 bytes 199 times --> 15.63 Mbps in 249.93 usec
39: 515 bytes 200 times --> 15.72 Mbps in 249.95 usec
40: 765 bytes 201 times --> 23.35 Mbps in 249.95 usec
41: 768 bytes 266 times --> 23.44 Mbps in 249.96 usec
42: 771 bytes 267 times --> 23.53 Mbps in 249.95 usec
43: 1021 bytes 134 times --> 31.17 Mbps in 249.93 usec
44: 1024 bytes 199 times --> 31.26 Mbps in 249.96 usec
45: 1027 bytes 200 times --> 31.35 Mbps in 249.95 usec
46: 1533 bytes 201 times --> 46.79 Mbps in 249.95 usec
47: 1536 bytes 266 times --> 46.88 Mbps in 249.96 usec
48: 1539 bytes 266 times --> 46.97 Mbps in 249.96 usec
49: 2045 bytes 134 times --> 62.44 Mbps in 249.88 usec
50: 2048 bytes 199 times --> 62.52 Mbps in 249.93 usec
51: 2051 bytes 200 times --> 62.60 Mbps in 249.95 usec
52: 3069 bytes 200 times --> 90.94 Mbps in 257.47 usec
53: 3072 bytes 258 times --> 93.77 Mbps in 249.96 usec
54: 3075 bytes 266 times --> 93.86 Mbps in 249.96 usec
55: 4093 bytes 133 times --> 0.79 Mbps in 39395.73 usec
56: 4096 bytes 3 times --> 130.85 Mbps in 238.83 usec
57: 4099 bytes 209 times --> 0.78 Mbps in 40288.30 usec
58: 6141 bytes 3 times --> 215.76 Mbps in 217.15 usec
59: 6144 bytes 307 times --> 213.06 Mbps in 220.00 usec
60: 6147 bytes 303 times --> 209.03 Mbps in 224.36 usec
61: 8189 bytes 148 times --> 249.97 Mbps in 249.94 usec
62: 8192 bytes 200 times --> 250.06 Mbps in 249.94 usec
63: 8195 bytes 200 times --> 250.14 Mbps in 249.95 usec
64: 12285 bytes 200 times --> 374.12 Mbps in 250.53 usec
65: 12288 bytes 266 times --> 375.07 Mbps in 249.95 usec
66: 12291 bytes 266 times --> 375.15 Mbps in 249.96 usec
67: 16381 bytes 133 times --> 500.02 Mbps in 249.94 usec
68: 16384 bytes 200 times --> 500.11 Mbps in 249.95 usec
69: 16387 bytes 200 times --> 495.25 Mbps in 252.44 usec
70: 24573 bytes 198 times --> 182.23 Mbps in 1028.81 usec
71: 24576 bytes 64 times --> 375.79 Mbps in 498.95 usec
72: 24579 bytes 133 times --> 370.36 Mbps in 506.33 usec
73: 32765 bytes 65 times --> 493.68 Mbps in 506.35 usec
74: 32768 bytes 98 times --> 500.58 Mbps in 499.42 usec
75: 32771 bytes 100 times --> 500.46 Mbps in 499.59 usec
76: 49149 bytes 100 times --> 542.02 Mbps in 691.82 usec
77: 49152 bytes 96 times --> 542.61 Mbps in 691.10 usec
78: 49155 bytes 96 times --> 542.38 Mbps in 691.44 usec
79: 65533 bytes 48 times --> 592.95 Mbps in 843.21 usec
80: 65536 bytes 59 times --> 590.32 Mbps in 846.99 usec
81: 65539 bytes 59 times --> 592.47 Mbps in 843.97 usec
82: 98301 bytes 59 times --> 11.43 Mbps in 65591.27 usec
83: 98304 bytes 3 times --> 277.76 Mbps in 2700.16 usec
84: 98307 bytes 24 times --> 15.06 Mbps in 49799.64 usec
85: 131069 bytes 3 times --> 510.80 Mbps in 1957.67 usec
86: 131072 bytes 25 times --> 514.21 Mbps in 1944.74 usec
87: 131075 bytes 25 times --> 510.50 Mbps in 1958.92 usec
88: 196605 bytes 25 times --> 546.13 Mbps in 2746.54 usec
89: 196608 bytes 24 times --> 546.14 Mbps in 2746.56 usec
90: 196611 bytes 24 times --> 546.09 Mbps in 2746.85 usec
91: 262141 bytes 12 times --> 572.33 Mbps in 3494.42 usec
92: 262144 bytes 14 times --> 572.34 Mbps in 3494.43 usec
93: 262147 bytes 14 times --> 572.22 Mbps in 3495.18 usec
94: 393213 bytes 14 times --> 570.56 Mbps in 5257.96 usec
95: 393216 bytes 12 times --> 572.23 Mbps in 5242.67 usec
96: 393219 bytes 12 times --> 572.40 Mbps in 5241.13 usec
97: 524285 bytes 6 times --> 572.65 Mbps in 6985.00 usec
98: 524288 bytes 7 times --> 572.46 Mbps in 6987.35 usec
99: 524291 bytes 7 times --> 571.93 Mbps in 6993.94 usec
100: 786429 bytes 7 times --> 573.59 Mbps in 10460.43 usec
101: 786432 bytes 6 times --> 571.90 Mbps in 10491.34 usec
102: 786435 bytes 6 times --> 571.60 Mbps in 10496.92 usec
103: 1048573 bytes 3 times --> 582.04 Mbps in 13744.67 usec
104: 1048576 bytes 3 times --> 579.27 Mbps in 13810.49 usec
105: 1048579 bytes 3 times --> 579.33 Mbps in 13808.99 usec
106: 1572861 bytes 3 times --> 573.60 Mbps in 20920.49 usec
107: 1572864 bytes 3 times --> 576.19 Mbps in 20826.32 usec
108: 1572867 bytes 3 times --> 571.19 Mbps in 21008.82 usec
109: 2097149 bytes 3 times --> 579.01 Mbps in 27633.51 usec
110: 2097152 bytes 3 times --> 574.73 Mbps in 27838.99 usec
111: 2097155 bytes 3 times --> 573.19 Mbps in 27914.00 usec
112: 3145725 bytes 3 times --> 571.46 Mbps in 41997.34 usec
113: 3145728 bytes 3 times --> 575.19 Mbps in 41725.52 usec
114: 3145731 bytes 3 times --> 577.54 Mbps in 41555.50 usec
115: 4194301 bytes 3 times --> 575.33 Mbps in 55620.51 usec
116: 4194304 bytes 3 times --> 574.36 Mbps in 55713.85 usec
117: 4194307 bytes 3 times --> 576.76 Mbps in 55482.32 usec
118: 6291453 bytes 3 times --> 577.00 Mbps in 83189.18 usec
119: 6291456 bytes 3 times --> 544.73 Mbps in 88117.50 usec
120: 6291459 bytes 3 times --> 402.50 Mbps in 119253.51 usec
121: 8388605 bytes 3 times --> 588.32 Mbps in 108784.51 usec
122: 8388608 bytes 3 times --> 588.85 Mbps in 108686.50 usec
123: 8388611 bytes 3 times --> 582.76 Mbps in 109823.15 usec

I found an NetPIPE for windowz also…

I repeated the NetPIPe test between winXP and suse.

The result of low bandwith with 4093 bytes and others is almost equal.
But the maximum bandwith stops at 333Mbps.

With linux to linux this is 580Mbps. I believe that is the limit of my pci bus.

Does anyone has a clue what is wrong?

Finally a solution.

Suse 10.0 is shipped with an old module for the nforce2 chipset witch controls the gigabit nic.

After updating suse with the linux driver from nvidia the problem is solved.