LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Distributions > Red Hat
User Name
Password
Red Hat This forum is for the discussion of Red Hat Linux.

Notices


Reply
  Search this Thread
Old 03-20-2017, 01:36 PM   #1
vijaymaurya
LQ Newbie
 
Registered: Jan 2017
Posts: 7

Rep: Reputation: Disabled
CPU I/O wait is getting increased with production Master DB server month by month


Hi All,

This is regarding the the production Master DB server, we are observing the CUP I/O wait is getting increased month by month.In 18-Dec-2016 it was 1 and now it has reach to 7 20-Mar-2017.This is a MySQL Database server which acting as a master in master-slave scenario.There is only one MySQL services is running on this server.I have checked with top, iotop,vmastat, iostat only the MySQL is the process utilizing some time high cpu resources not every regularly.

Please find the output of iostat command.

iostat
Linux 2.6.18-419.el5 (IFweb03.ifglobal.imagefirst.com) 03/20/2017

avg-cpu: %user %nice %system %iowait %steal %idle
10.69 0.00 2.30 6.76 0.00 80.25

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 2.11 711.56 0.01 542013204 9962
sda1 2.10 708.56 0.01 539733732 9962
sdb 221.81 524.78 3884.15 399739502 2958670592
sdb1 194.42 392.76 3672.90 299177871 2797753512
sdb2 17.49 34.33 150.66 26147479 114761744
sdb3 9.71 95.91 60.21 73058750 45862864
sdb4 0.00 0.00 0.00 11 0
sdb5 0.19 1.78 0.38 1353544 292472
sdc 5.37 10.25 45.12 7809251 34366192
sdc1 5.37 10.25 45.12 7808084 34366192
sdd 58.33 1.60 1429.52 1221881 1088904368
sdd1 58.33 1.60 1429.52 1220714 1088904368
dm-0 687.50 536.61 5358.78 408747969 4081940984
dm-1 0.00 0.01 0.00 4167 280

Please find the output of vmstat command.

vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 136 4635948 182124 6687316 0 0 156 670 7 13 11 2 80 7 0



Please find the below output of top command.

top - 14:32:12 up 8 days, 19:34, 1 user, load average: 1.31, 0.89, 0.98
Tasks: 138 total, 1 running, 137 sleeping, 0 stopped, 0 zombie
Cpu(s): 22.4%us, 2.6%sy, 0.0%ni, 59.4%id, 13.5%wa, 0.6%hi, 1.6%si, 0.0%st
Mem: 17831692k total, 13175740k used, 4655952k free, 182016k buffers
Swap: 5898232k total, 136k used, 5898096k free, 6672088k cached


Note:I have observed that both OS and MySQL data directory is mounted on same partition using LVM. Can this might be cause in increasing the i/o wait as both os and mysql reading and writing on same disk.if i add new disk and mount the MySQL data directory to that partition. will it help us to decrease the CPU I/O WAIT.

Thanks in advance.
 
Old 03-21-2017, 10:37 PM   #2
dijetlo
Senior Member
 
Registered: Jan 2009
Location: RHELtopia....
Distribution: Solaris 11.2/Slackware/RHEL/
Posts: 1,491
Blog Entries: 2

Rep: Reputation: Disabled
Hi Vijay,
Welcome to linux questions.
Code:
 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0    140 1177292 413928 4837840    0    0     1    16    6   18  5  1 95  0  0
(root) bash-4.4$
It would be helpful if when you post output from commands like vmstat or iostat, you put it in code brackets (see "advanced options"). It preserves formatting making what you post much easier for us to read (see above).

Question:
Are you sitting on virtualized infrastructure and if so, what kind? Is it clustered ? Are you using a san/nas? If so, is it virtualized as well or physical?
The other option is you're sitting in bare metal on local disk. Different kinds of storage behave differently and those numbers can mean different things (or nothing at all) depending on the virtualization platform you're using.

The other question is, was your master db growing over this period ? IOWAIT includes things like "seek", Without knowing your storage framework, this question
Quote:
.if i add new disk and mount the MySQL data directory to that partition. will it help us to decrease the CPU I/O WAIT.
Would be little more than a guess ( in which case, no, probably not, LVM abstracts most of what you think of as "disk", though you might have a chokepoint at the controller).

Thanks

Last edited by dijetlo; 03-21-2017 at 10:50 PM.
 
Old 03-22-2017, 12:08 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 15,418

Rep: Reputation: 2013Reputation: 2013Reputation: 2013Reputation: 2013Reputation: 2013Reputation: 2013Reputation: 2013Reputation: 2013Reputation: 2013Reputation: 2013Reputation: 2013
And I was in agreement till that edit @dijetlo.

LVM still has to do the physical I/O down to the pv - I always prefer isolation where it is available. Also helps make the job of determining who's actually doing harm (if any) to whom a little easier.
Of itself, iowait isn't really an indicator of what most people think of - but it might be. Simply spreading the load might help.
 
Old 03-22-2017, 04:02 PM   #4
dijetlo
Senior Member
 
Registered: Jan 2009
Location: RHELtopia....
Distribution: Solaris 11.2/Slackware/RHEL/
Posts: 1,491
Blog Entries: 2

Rep: Reputation: Disabled
Quote:
Also helps make the job of determining who's actually doing harm (if any) to whom a little easier.
I agree with you entirely, if one or both of us were standing there working on it, however this a message board and I don't want to send the guy down a rabbit hole for no reason.
What I was thinking is if he is getting congestion around the controller on a DB node, just moving the DB to another disk isn't going to be extremely helpful, it isn't as if the compute node is contributing much to the IO issue in normal operation, it's all being driven by the DB. He'd just have the same problem on a new disk.
He'd have to shard the DB across multiple disks to break that up but that's an even deeper rabbit hole to send him down, I think we'd have to understand his configuration (it could just be he's on a cluster that doesn't support async i/o and what he's seeing is the normal operation of the Cluster FS or he's on bare metal with a wonky HD or <insert your favorite storage horror story here>
Still, you're points well taken, Syg00, thanks

Last edited by dijetlo; 03-22-2017 at 04:06 PM.
 
Old 03-23-2017, 12:50 PM   #5
vijaymaurya
LQ Newbie
 
Registered: Jan 2017
Posts: 7

Original Poster
Rep: Reputation: Disabled
Hi dijetlo,

Thanks for your reply.

Please find the output of vmstat and iostat.

Code:
vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  1    136 6129920 167332 5318088  0    0   164   696    7    2 11  3 79  7  0

iostat
Linux 2.6.18-419.el5 (IFweb03.ifglobal.imagefirst.com)  03/23/2017

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.38    0.00    2.56    7.06    0.00   79.00

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               2.10       711.63         0.01  724522092      10006
sda1              2.09       708.64         0.01  721474972      10006
sdb             234.94       587.85      4057.25  598498510 4130726088
sdb1            203.75       431.98      3816.65  439806655 3885771496
sdb2             13.76        39.00       116.19   39707951  118292960
sdb3             17.21       114.91       123.91  116995254  126153880
sdb4              0.00         0.00         0.00         11          0
sdb5              0.22         1.95         0.50    1986792     507752
sdc               6.26        12.56        52.35   12784707   53299304
sdc1              6.26        12.56        52.35   12783540   53299304
sdd              61.88         1.79      1457.14    1822369 1483530472
sdd1             61.88         1.79      1457.14    1821202 1483530472
dm-0            715.49       602.18      5566.75  613082921 5667555584
dm-1              0.00         0.00         0.00       4167        280
Are you sitting on virtualized infrastructure and if so, what kind? we are using VMware ESXi 5.1

Is it clustered ? No.

If so, is it virtualized as well or physical? it is virtualized.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] rsnapshot - missing last month on a 3-month cycle. Fedora 15 steve_s Linux - Software 2 02-17-2012 04:51 AM
LXer: Pandora Handheld To Finally Go Into Mass Production Next Month LXer Syndicated Linux News 1 01-28-2012 02:10 AM
LXer: CodeWeavers CEO names himself employee of the month for 175TH consecutive month LXer Syndicated Linux News 3 08-13-2010 04:55 PM
transform month number to month name in php ALInux Programming 1 11-09-2005 10:45 AM
Should I wait a month to configure this machine? Pcghost Linux - Distributions 0 04-01-2004 06:17 PM


All times are GMT -5. The time now is 01:15 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration