LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 06-21-2009, 10:21 PM   #1
john.daker
Member
 
Registered: Jul 2008
Posts: 33

Rep: Reputation: 15
Dual core cpu, 2 scp processes, 100% iowait.


Dear all ,

I found that I have gotten a high IOwait in `mpstat` when has two scp processes coping large(>500G) dir from another two different machines.

cpu: Intel(R) Xeon(R) CPU 3060 @ 2.40GHz
os: Red Hat Enterprise Linux AS release 4 (Nahant Update 3)
raid 5, only 2 scp processes , not errors in /var/log/messages

load average: 10.89, 9.75, 9.11
mpstat:
Code:
11:09:37 AM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
11:09:39 AM  all    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1235.20
11:09:39 AM    0    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1235.20
11:09:39 AM    1    0.00    0.00    0.00  100.00    0.00    0.00    0.00      0.00

11:09:39 AM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
11:09:41 AM  all    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1351.28
11:09:41 AM    0    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1351.79
11:09:41 AM    1    0.00    0.00    0.00  100.51    0.00    0.00    0.00      0.00

11:09:41 AM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
11:09:43 AM  all    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1285.20
11:09:43 AM    0    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1284.69
11:09:43 AM    1    0.00    0.00    0.00   99.49    0.00    0.00    0.00      0.00

11:09:43 AM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
11:09:45 AM  all    0.00    0.00    0.51   99.49    0.00    0.00    0.00   1356.12
11:09:45 AM    0    0.00    0.00    0.51   99.49    0.00    0.00    0.00   1356.12
11:09:45 AM    1    0.00    0.00    0.00  100.00    0.00    0.00    0.00      0.00

11:09:45 AM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
11:09:47 AM  all    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1379.08
11:09:47 AM    0    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1379.08
11:09:47 AM    1    0.00    0.00    0.00  100.00    0.00    0.00    0.00      0.00

11:09:47 AM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
11:09:49 AM  all    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1292.82
11:09:49 AM    0    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1292.82
11:09:49 AM    1    0.00    0.00    0.00  100.51    0.00    0.00    0.00      0.00
iostat -x 1:
Code:
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0   0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
cciss/c0d1   0.00   0.00  0.00 206.12    0.00   24.49     0.00    12.24     0.12  1097.90 60859.30   4.96 102.14

avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.00    0.00    0.00   50.00   50.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0   0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
cciss/c0d1   0.00   0.00  0.00 156.12    0.00   16.33     0.00     8.16     0.10   911.27 57661.75   6.55 102.24
And my system is very very slow , a `ls` would be 5~6 seconds...

Any idea of it?

Thanks a lot.

Last edited by john.daker; 06-21-2009 at 11:03 PM.
 
Old 06-21-2009, 11:45 PM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,625

Rep: Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088Reputation: 2088
Better hope you don't have any other (particularly system) files that you are trying to access at the same time on that disk.
And all those interrupts need to be handled - in interrupt context. This doesn't show anywhere in CPU%.

iowait merely means there is I/O outstanding, and there wasn't anything actually running on the CPUs when the sampling was done. Think of it as 100% idle with I/O outstanding.
 
Old 06-22-2009, 04:09 AM   #3
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 4,062

Rep: Reputation: 893Reputation: 893Reputation: 893Reputation: 893Reputation: 893Reputation: 893Reputation: 893
Quote:
Originally Posted by john.daker View Post
Dear all ,

I found that I have gotten a high IOwait in `mpstat` when has two scp processes coping large(>500G) dir from another two different machines.
Errr, yes, well it would. Assuming that you actually wanted to copy the files, you'll have to copy the files at some time. If you weren't in a hurry, you could try adjusting the nice value, but that would probably make the problem less severe but last longer, so its unclear that would be a real advantage. And, the files have probably copied by now, so i am not clear what you want to do about it now.


Quote:
mpstat:
Code:
11:09:37 AM  CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s
11:09:39 AM  all    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1235.20
11:09:39 AM    0    0.00    0.00    0.00  100.00    0.00    0.00    0.00   1235.20
11:09:39 AM    1    0.00    0.00    0.00  100.00    0.00    0.00    0.00      0.00
....
anytime you see an iowait of 100%, you can't have much time to do anything else

Quote:
iostat -x 1:
Code:
avg-cpu:  %user   %nice    %sys %iowait   %idle
           0.00    0.00    0.00   50.00   50.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0   0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
cciss/c0d1   0.00   0.00  0.00 156.12    0.00   16.33     0.00     8.16     0.10   911.27 57661.75   6.55 102.24
I'm guessing that iostat and mpstat handle iowait (and, presumably the other reported times) differently; how many cores do you have?

Quote:
And my system is very very slow , a `ls` would be 5~6 seconds..
Seems reasonable, under the circumstances.

Quote:
Any idea of it?
I'm not quite sure what you want to do about it; not copying the files, or not copying the files at a time when the system is required to do anything else (midnight cron job??) would be the easiest, but presumably don't meet some other requirement.
 
Old 06-22-2009, 04:18 AM   #4
vap16oct1984
Member
 
Registered: Jun 2009
Location: INDIA
Distribution: RHEL-5
Posts: 174
Blog Entries: 3

Rep: Reputation: 38
hey how much memory do u have in ur system. pls show me the memory status
 
Old 06-22-2009, 08:16 AM   #5
john.daker
Member
 
Registered: Jul 2008
Posts: 33

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by salasi View Post
Errr, yes, well it would. Assuming that you actually wanted to copy the files, you'll have to copy the files at some time. If you weren't in a hurry, you could try adjusting the nice value, but that would probably make the problem less severe but last longer, so its unclear that would be a real advantage. And, the files have probably copied by now, so i am not clear what you want to do about it now.
...
Yes, it's a cron job, it copies two dirs from two different machines with scp, and need scp at same time...

it's dual-core, 4GB RAM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Single cpu-core used on multicore system when processes communicate with pipe merijnv Linux - Kernel 3 04-10-2009 09:17 PM
Can you install two OS on a dual-core computer (one OS on each CPU core?) kushalkoolwal Linux - Software 7 10-17-2008 05:57 AM
Can two processes run at the same time on a dual-core processor? exceed1 Linux - General 11 04-08-2008 11:19 PM
High Load, Low CPU/RAM/iowait ? newlinuxnewbie Linux - Server 1 09-22-2006 09:24 AM
High loadavg, low CPU & iowait reichhartkg Linux - General 1 07-28-2006 03:44 PM


All times are GMT -5. The time now is 08:57 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration