LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware
User Name
Password
Linux - Hardware This forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?

Notices

Reply
 
Search this Thread
Old 02-07-2013, 07:12 AM   #1
diwljina
Member
 
Registered: Jun 2009
Distribution: Slackware, Debian
Posts: 105

Rep: Reputation: 8
Disk iowait too big when copying


Hi!

I have dedicated server with 3 HDD. System disk, backup disk (same as system disk) and data disk. When I copy lot of data with cp (say, between backup disk and data disk) load average goes sky high.

Load average at the moment: 0.57
When copying: 50 and more

Copying with rsync and with --bwlimit=10000 goes without a problem. Higher values cause high load.

File system is ext3.

sda - system disk
Code:
hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   13444 MB in  2.00 seconds = 6730.82 MB/sec
 Timing buffered disk reads:  232 MB in  3.02 seconds =  76.73 MB/sec
sdb - data disk
Code:
hdparm -Tt /dev/sdb

/dev/sdb:
 Timing cached reads:   13740 MB in  2.00 seconds = 6879.30 MB/sec
 Timing buffered disk reads:  430 MB in  3.00 seconds = 143.10 MB/sec
sdc - backup disk
Code:
hdparm -Tt /dev/sdc

/dev/sdc:
 Timing cached reads:   13796 MB in  2.00 seconds = 6907.75 MB/sec
 Timing buffered disk reads:  336 MB in  3.01 seconds = 111.45 MB/sec
iostat
Code:
Linux 2.6.18-308.20.1.el5PAE (host)        02/07/2013

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.90    0.15    3.21    3.72    0.00   86.02

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda             102.24       515.58      1417.08 1233146404 3389300222
sda1              0.00         0.18         0.00     427634         14
sda2              7.40       175.21        67.31  419065023  160994320
sda3             58.95        45.64      1361.83  109156607 3257161312
sda4              0.00         0.00         0.00          6          0
sda5              0.80        19.04         7.58   45538562   18121096
sda6              0.18         2.34         7.31    5587138   17493648
sda7             30.79       269.43      1266.51  644403856 3029179912
sda8              4.11         3.75       502.27    8965250 1201317216
sdb               2.83        61.13       268.24  146216772  641567958
sdb1              2.83        61.13       268.24  146211544  641567950
sdc               2.12       107.77       147.37  257760600  352482448
sdc1              2.12       107.77       147.37  257758576  352482448

iostat -x 1 (when not copying)
Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.74    0.00    2.74    0.75    0.00   91.77

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    71.00  5.00 24.00   400.00   760.00    40.00     0.19    6.48   1.21   3.50
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda3              0.00    14.00  0.00  3.00     0.00   136.00    45.33     0.00    0.33   0.33   0.10
sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda7              0.00    57.00  5.00 21.00   400.00   624.00    39.38     0.19    7.19   1.31   3.40
sda8              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
iostat -x 1 (when copying: sdc > sdb)
Code:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          14.21    0.00   11.47   54.11    0.00   20.20

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              29.00   104.00 69.00 76.00  1272.00  1664.00    20.25    35.83  287.32   6.91 100.20
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     4.00  2.00 20.00    16.00   192.00     9.45    14.42  698.91  45.55 100.20
sda3              0.00     2.00 44.00  3.00   352.00    40.00     8.34     2.42   63.77  21.32 100.20
sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda5              0.00     4.00  0.00  1.00     0.00    40.00    40.00     0.15  137.00 146.00  14.60
sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda7             29.00     0.00 23.00 22.00   904.00   240.00    25.42     8.86  326.62  22.27 100.20
sda8              0.00    94.00  0.00 30.00     0.00  1152.00    38.40     9.99  281.77  33.40 100.20
sdb               0.00  9352.00  0.00 145.00     0.00 128000.00   882.76   129.16 1087.48   6.91 100.20
sdb1              0.00  9352.00  0.00 145.00     0.00 128000.00   882.76   129.16 1087.48   6.91 100.20
sdc             580.00     0.00 332.00  0.00 134048.00     0.00   403.76     0.64    1.92   1.84  61.20
sdc1            580.00     0.00 332.00  0.00 134048.00     0.00   403.76     0.64    1.92   1.84  61.20

Code:
cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
Other two disks are "dedline" now, but they were "cfq" as well. Just tried to see if there will be any difference. There isn't.

Any operations that are more disk intensive are killing the server. If some process uses more memory and there is need for swaping, load goes very high. Sometimes I have to kill some service so load can go down. There were times when load went to 500 because of swaping. Server has 4GB RAM and Xeon X3220 @ 2.40GHz. I can accept poor performance when there is not enough RAM, but just copying shouldn't kill the server. This just doesn't seams right.

Any idea what could be the problem? What else should I check? Could it be bad motherboad controller?

Added:
What I would like to know is: is this really normal? Can it be that it is hardware problem and I can request another machine from provider?

Last edited by diwljina; 02-08-2013 at 08:13 AM. Reason: Added question.
 
Old 02-08-2013, 04:37 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,035

Rep: Reputation: 968Reputation: 968Reputation: 968Reputation: 968Reputation: 968Reputation: 968Reputation: 968Reputation: 968
Better find out what is hitting that /dev/sda. Looks like it might be on the same controller as /dev/sdb, but why all the write traffic ?. Try this when the loadavg gets high
Code:
top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
More up to date kernel would get better data/tools.
 
Old 02-09-2013, 06:10 AM   #3
diwljina
Member
 
Registered: Jun 2009
Distribution: Slackware, Debian
Posts: 105

Original Poster
Rep: Reputation: 8
sda2 = /usr
Apache is on /usr partition and logs directory is there as well. I guess that's what hitting it.

sda3 = /var
Aside from system log, I tried to see what else is used during high load.
Code:
% lsof +D /var
/var/run/dovecot
... (other dovecot related files like login)
/var/named/
... (other named related stuff)
/var/spool/exim
/var/log/exim_mainlog
... (some other stuff that I don't think are unusual)
php       20741 username  mem    REG        8,3      5120   1579829 /var/cache/fontconfig/87f5e051180a7a45f16eb6fe7dbd3749-x86.cache-2
... (about 50 lines like this one)
sda7 = /home
mysql is there and couple of websites. I expect files to be written here by website users (uploads of files ~2-20MB).

sda8 = /tmp
lsof +D shows nothing unusual. Some pid and log files. Nothing big or intensive.

But iowait on sda goes up only when copying, even though it's from sdc to sdb.


Output of your command:
Code:
top - 06:02:26 up 29 days, 15:36,  5 users,  load average: 24.89, 15.65, 11.88
Tasks: 415 total,   2 running, 411 sleeping,   0 stopped,   2 zombie
Cpu(s):  6.8%us,  2.6%sy,  0.1%ni, 86.3%id,  3.7%wa,  0.1%hi,  0.4%si,  0.0%st
Mem:   4150504k total,  4018128k used,   132376k free,    37796k buffers
Swap:  2096440k total,   118828k used,  1977612k free,  2477104k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                 
13949 root      16   0 72744  32m  896 D 34.9  0.8   0:18.56 rsync                                   
  219 root      10  -5     0    0    0 D  3.9  0.0  13:22.82 kswapd0                                 
16579 root      15   0     0    0    0 D  1.9  0.0   0:00.06 pdflush                                 
18664 root      10  -5     0    0    0 D  1.9  0.0   2:39.47 kjournald                               
 1635 root      10  -5     0    0    0 D  0.0  0.0   3:38.09 kjournald                               
 1637 root      10  -5     0    0    0 D  0.0  0.0  13:39.84 kjournald                               
 1641 root      10  -5     0    0    0 D  0.0  0.0  89:54.68 kjournald                               
 1643 root      10  -5     0    0    0 D  0.0  0.0   1:32.68 kjournald                               
 2780 root      16   0  1824  500  448 D  0.0  0.0   1:40.31 syslogd                                 
11562 nobody    16   0  184m  21m  10m D  0.0  0.5   0:00.21 httpd                                   
15153 user1  15   0  164m  10m 7112 D  0.0  0.3   0:00.07 php                                     
15167 user2  18   0 39992  12m 6916 D  0.0  0.3   0:00.40 php                                     
15251 mailnull  18   0  9380 1652 1284 D  0.0  0.0   0:00.12 exim                                    
15322 root      18   0  4940  552  484 D  0.0  0.0   0:00.51 md5sum                                  
15555 nobody    15   0  174m  15m 5300 D  0.0  0.4   0:00.01 httpd                                   
15798 nobody    15   0  175m  16m 5320 D  0.0  0.4   0:00.08 httpd                                   
15974 user1  15   0  164m  10m 7108 D  0.0  0.3   0:00.10 php                                     
16005 user2  18   0 21520  676  444 D  0.0  0.0   0:00.00 php                                     
16050 nobody    15   0  174m  14m 3800 D  0.0  0.3   0:00.00 httpd                                   
16220 nobody    15   0  174m  13m 3176 D  0.0  0.3   0:00.00 httpd                                   
16371 mailnull  18   0  9388 2260 1828 D  0.0  0.1   0:00.00 exim                                    
16459 nobody    15   0  174m  14m 4212 D  0.0  0.4   0:00.00 httpd                                   
16506 mailnull  18   0  9388 2260 1836 D  0.0  0.1   0:00.00 exim                                    
16571 root      15   0     0    0    0 D  0.0  0.0   0:00.11 pdflush                                 
16592 mailnull  18   0  9384 1792 1404 D  0.0  0.0   0:00.00 exim                                    
16594 mailnull  18   0  9384 2168 1756 D  0.0  0.1   0:00.00 exim                                    
16634 root      18   0  2172  200  164 D  0.0  0.0   0:00.00 awk                                     
Total status D: 27
 
Old 02-09-2013, 08:52 AM   #4
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 1,978

Rep: Reputation: 512Reputation: 512Reputation: 512Reputation: 512Reputation: 512Reputation: 512
Where is swap?

One thing to watch, as you increase the I/O bandwidth, it also needs more memory for buffers to maintain that rate. Then you block on input while the buffer gets dumped. This could impair/defeat overlapped I/O to maintain a total throughput.

You can check how paging activity may be involved using "vmstat -a 5" for a 5 second delay between samples.
 
Old 02-09-2013, 10:54 AM   #5
diwljina
Member
 
Registered: Jun 2009
Distribution: Slackware, Debian
Posts: 105

Original Poster
Rep: Reputation: 8
swap is on sda6.

Interesting thought about more memory needed during more intensive I/O operations. Swaping does load system. What process is responsible for buffering? Should I see it in ps output?

Code:
% vmstat -a 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 3 11 119316 138284 3002440 873676    0    1    94    32    1    0  7  3 86  4  0
 2 10 119316 140028 3001864 873132    0    0 33310 10364 1847 6106 19  6 11 64  0
 0 19 119280 138736 2996260 873484    7    0 10847 31626 1531 2951  4  3 16 77  0
 1 11 119272 139188 3016032 860512   72    0 34753 20876 1758 6460  8  6 14 72  0
 0 12 119272 133896 3006924 865364   11    0 24936 22926 1666 4939  6  5 20 69  0
This is during copying. Load is 19.36 and going up.
 
Old 02-09-2013, 12:24 PM   #6
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 1,978

Rep: Reputation: 512Reputation: 512Reputation: 512Reputation: 512Reputation: 512Reputation: 512
There are several places responsible for buffering - the kernel itself, and the rsync trying to package the data for transfer (it reads into somewhere, then writes). I would expect rsync itself uses multiple buffers to attempt to increase throughput.

Swap in isn't so bad (one way only). It can be a problem when both swap in/out occur. The log you show shouldn't be a problem as there is plenty of memory for buffers.
 
1 members found this post helpful.
Old 02-09-2013, 01:30 PM   #7
diwljina
Member
 
Registered: Jun 2009
Distribution: Slackware, Debian
Posts: 105

Original Poster
Rep: Reputation: 8
So can I assume that syg00's right, about sda being on same controller as sdb? Could it be the real problem? Will going with hardware RAID card solve the problem? RAID10 maybe?
 
Old 02-09-2013, 05:01 PM   #8
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,035

Rep: Reputation: 968Reputation: 968Reputation: 968Reputation: 968Reputation: 968Reputation: 968Reputation: 968Reputation: 968
You really don't want to see things like this
Code:
  219 root      10  -5     0    0    0 D  3.9  0.0  13:22.82 kswapd0                                 
16579 root      15   0     0    0    0 D  1.9  0.0   0:00.06 pdflush
The status "D" says uninterruptible sleep - in this case waiting for (disk) I/O. kjournald is (always) a bad sign when it shows up too.

I'd be looking at the hardware. May not be just the controller - if the data is funnelled down the same (slow) bus you could find that is your bottle-neck. Can't really be diagnosed at a distance, just a matter of testing.
Also, as I said, a (much) more current kernel will help with things like pdflush.

Last edited by syg00; 02-09-2013 at 05:02 PM.
 
1 members found this post helpful.
Old 02-10-2013, 05:35 AM   #9
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 1,978

Rep: Reputation: 512Reputation: 512Reputation: 512Reputation: 512Reputation: 512Reputation: 512
What distribution are you using?

I ask because Fedora has implemented a use of cgroups that can cause issues when copying large files - it can force a CPU thrashing against a process in its attempt at fair share scheduling. It is a "works mostly" situation - it appears to help interactivity by focusing the CPU on interactive activity, but it penalizes things that are CPU/memory bound.
 
Old 02-10-2013, 10:05 AM   #10
diwljina
Member
 
Registered: Jun 2009
Distribution: Slackware, Debian
Posts: 105

Original Poster
Rep: Reputation: 8
Quote:
Originally Posted by syg00 View Post
You really don't want to see things like this
Code:
  219 root      10  -5     0    0    0 D  3.9  0.0  13:22.82 kswapd0                                 
16579 root      15   0     0    0    0 D  1.9  0.0   0:00.06 pdflush
The status "D" says uninterruptible sleep - in this case waiting for (disk) I/O. kjournald is (always) a bad sign when it shows up too.

I'd be looking at the hardware. May not be just the controller - if the data is funnelled down the same (slow) bus you could find that is your bottle-neck. Can't really be diagnosed at a distance, just a matter of testing.
Also, as I said, a (much) more current kernel will help with things like pdflush.
I do have those processes on top of iotop output during copying and other intensive I/O operations:
Code:
Total DISK READ: 12.31 M/s | Total DISK WRITE: 52.82 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                 
 1637 be/3 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [kjournald]
 1641 be/3 root        0.00 B/s    0.00 B/s 99.99 % 99.99 % [kjournald]
18664 be/3 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [kjournald]
 8788 be/4 mailnull  144.90 K/s    0.00 B/s  0.00 % 98.61 % exim -bpu
 9522 be/4 mailnull  114.40 K/s    0.00 B/s  0.00 % 96.27 % exim -bpc
 9544 be/4 root        0.00 B/s   72.45 K/s  0.00 % 91.60 % [pdflush]
 9301 be/4 root        0.00 B/s   49.57 K/s  0.00 % 86.29 % [pdflush]
 5984 be/4 root       10.77 M/s    0.00 B/s  0.00 % 70.88 % rsync -av --progress ~html/var/tmp
  219 be/3 root        0.00 B/s    0.00 B/s  0.00 % 35.30 % [kswapd0]
 6022 be/4 nobody      3.81 K/s    0.00 B/s  0.00 % 30.39 % httpd -k start -DSSL
 9594 be/4 mailnull  129.65 K/s    0.00 B/s  0.10 % 21.20 % exim -Mc 1U4Z6j-0002Tl-CQ
 5986 be/4 root       19.07 K/s   10.54 M/s 70.88 % 19.76 % rsync -av --progress ~html/var/tmp
 9547 be/4 nobody     15.25 K/s    0.00 B/s 96.27 %  6.79 % httpd -k start -DSSL
 9548 be/4 nobody      7.63 K/s    0.00 B/s 91.60 %  5.60 % httpd -k start -DSSL
 8271 be/4 nobody      7.63 K/s    0.00 B/s  0.00 %  5.30 % httpd -k start -DSSL
 8848 be/4 nobody      7.63 K/s    0.00 B/s  0.00 %  4.39 % httpd -k start -DSSL
...
I will probably get new hardware and will install newer kernel.
Quote:
Originally Posted by jpollard
What distribution are you using?

I ask because Fedora has implemented a use of cgroups that can cause issues when copying large files - it can force a CPU thrashing against a process in its attempt at fair share scheduling. It is a "works mostly" situation - it appears to help interactivity by focusing the CPU on interactive activity, but it penalizes things that are CPU/memory bound.
It's Red Hat Enterprise Linux Server release 5.8. I guess that kind of new things from Fedora didn't get to be in RedHat yet.
 
Old 02-22-2013, 10:51 AM   #11
diwljina
Member
 
Registered: Jun 2009
Distribution: Slackware, Debian
Posts: 105

Original Poster
Rep: Reputation: 8
Replaced cables and then whole server with the same one (controllers are onboard). Nothing changed. Motherboard is SuperMicro PDSMU. Guess it's this one:
http://www.supermicro.com/products/m...3010/pdsmu.cfm

I conclude that motherboard was not malfunctioning, it just wasn't fast enough.

It is using Intel® ICH7R SATA2 (3 Gbps) controller built-in. Is it really that slow to expect this behavior? It's 3 Gbps. Can I expect hardware RAID card to improve situation?

I mounted all partitions with noatime. Is there anything else I can do to decrease disk operations?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
problem after copying 250GB disk to 1TB disk (cannot resize the old EXTENDED) pabelmont Linux - Newbie 5 02-04-2013 08:38 AM
Big disk - big partition Thor_2.0 Linux - Hardware 6 12-13-2008 02:54 PM
error while copying a big size folder dissident_goodchild Suse/Novell 10 08-28-2007 05:56 PM
high IOWait on server when copying files from network InDubio Linux - Networking 2 03-12-2007 03:10 AM
Copying big files in RAID5 will stall system Micro420 Linux - Hardware 1 03-07-2007 08:34 PM


All times are GMT -5. The time now is 07:14 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration