Low actual disk throughput over network

squisher · 02-10-2007, 02:22 PM

Hi everyone,
my server somehow gives me much slower disk I/O speed when I use it over the network than I would expect. I've had this problem for a while now and haven't made any progress, so I hope maybe someone here has an idea. Let me know if you need any additional information and I'll post them.

I have a P3 933MHz server with 512MB RAM, and a Hitachi 250GB (8mb cache) harddrive. It's running Debian testing with stock kernel (2.6.18-3-686), but I ave also tested it with 2.6.12-1-386 (which gave me similar results).

hdparm:

Code:

/dev/hda:
 multcount    = 16 (on)
 IO_support   =  1 (32-bit)
 unmaskirq    =  0 (off)
 using_dma    =  1 (on)
 keepsettings =  0 (off)
 readonly     =  0 (off)
 readahead    = 256 (on)
 geometry     = 30401/255/63, sectors = 488397168, start = 0

The local disk performance tests show good results:
hdparm -t -T /dev/hda:

Quote:

/dev/hda:
Timing cached reads: 226 MB in 2.00 seconds = 112.94 MB/sec
Timing buffered disk reads: 154 MB in 3.01 seconds = 51.13 MB/sec

bonnie++:
Sequential Output:

Code:

Per Char: 15062	K, 97% CPU
Block:    45628 K, 60% CPU
Rewrite:  25316 K, 32% CPU

Sequential Input:

Code:

Per Char: 11275 K, 72% CPU
Block:    55849 K, 43% CPU

So far so good, but since this is a server, I mainly use the disk over the network. I tested with netcat and iperf I do get speeds up to 28 MB/s (this is gigabit).

However, when I download a big (700MB) file over http I only get around 11 MB/s, and over nfs only 9 MB/s.

Here's some vmstat output, where the first two rows are before the transfer starts:
http:

Code:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  1     12 195984  20660 112540    0    0   112    82  181  250  5  1 93  1
 0  0     12 195860  20776 112544    0    0     0    94 1764  538  1  2 97  0
 0  1     12 163340  20876 144848    0    0  6443    55 3640  684  3 28 43 26
 0  1     12 108284  21004 199772    0    0 10998    22 4992  720  1 46  1 51
 0  1     12  53584  21112 254324    0    0 10921    20 5064  735  2 47  0 51
 0  0     12   7792  21280 299852    0    0  9117    37 4324  688  1 39 21 39
 1  1     12   5836  17892 305396    0    0  8938    10 4282  704  1 38 22 39
 0  1     12   5816  15216 307696    0    0 11413     6 5010  735  1 49  1 49
 0  1     12   5812  15200 307884    0    0  9901    46 4551  758  3 42  1 54
 1  0     12   6184  15248 307388    0    0 10726    20 4811  752  1 45  0 53
 1  1     12   5608  15280 308000    0    0 11278    12 4965  760  1 48  4 47
... [truncated] ...

nfs:

Code:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  1     12  10940  15900 302756    0    0   117    82  188  252  5  1 92  1
 0  0     12  10460  15968 303164    0    0    81    36 1128  557  1  1 97  1
 0  8     12   6056  15920 308948    0    0  8844    19 3831  962  4 39 21 35
 0  8     12   6332  15612 309204    0    0 10802    25 4482 1075  2 49  0 50
 3  8     12   6456  15560 309048    0    0 10811   242 4408 1041  3 47  2 48
 0  8     12   6352  15556 308684    0    0  8978   146 3869  980  3 40  0 57
 0  8     12   5492  11424 313908    0    0 10596    24 4413 1076  3 47  0 50
 0  8     12   5720   9728 315512    0    0 10628    32 4404 1048  3 48  0 49
 0  8     12   6052   9712 314876    0    0  6467    10 3115  893  4 30 33 34
 0  8     12   6052   9632 315096    0    0 10738    41 4408 1062  2 47  0 51
 0  8     12   5964   8980 315956    0    0  9662     9 4127 1044  2 44  0 54
 0  9     12   5812   8344 316768    0    0 10373     4 4255 1061  3 46  2 49
 0  9     12   6424   8112 316476    0    0  7799    10 3550  962  2 35  9 53
... [truncated] ...

Why is the CPU spending so much time in system and waiting? The network stack can't be introducing such a huge performance hit. It's a good network card (intel e1000), and over loopback the CPU is fast enough to get 1 Gbit/s.

Any help on how I can speed things up would be greatly appreciated!

jailbait · 02-11-2007, 01:27 PM

"However, when I download a big (700MB) file over http I only get around 11 MB/s, and over nfs only 9 MB/s."

Any computer action runs at the speed of the slowest component involved. When you are doing your local tests you are running at the speed of your disk I/O. when you are doing your download tests you are running at the speed of your network connection.

"Why is the CPU spending so much time in system and waiting?"

The CPU only has to do enough work to keep up with the speed of the slowest component. The CPU is spending more time waiting for a network connection than a disk copy because the network is slower than a disk.

--------------------
Steve Stites

squisher · 02-11-2007, 01:40 PM

Quote:

Originally Posted by jailbait

"However, when I download a big (700MB) file over http I only get around 11 MB/s, and over nfs only 9 MB/s."

Any computer action runs at the speed of the slowest component involved. When you are doing your local tests you are running at the speed of your disk I/O. when you are doing your download tests you are running at the speed of your network connection.

My network connection is gigabit, and as I tested with both netcat and iperf that that is not the bottleneck. I do get up to 28 MB/s out of my network.

Quote:

Originally Posted by jailbait

"Why is the CPU spending so much time in system and waiting?"

The CPU only has to do enough work to keep up with the speed of the slowest component. The CPU is spending more time waiting for a network connection than a disk copy because the network is slower than a disk.

Yes, I understand that, but:
Disk Speed: ~50 MB/s
Network Speed: ~28 MB/s

So why can I download faster than 11 MB/s? That is my problem.