Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum. |
| Notices |
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
Are you new to LinuxQuestions.org? Visit the following links:
Site Howto |
Site FAQ |
Sitemap |
Register Now
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
|
 |
04-02-2014, 12:32 PM
|
#1
|
|
Member
Registered: Jun 2008
Posts: 122
Rep: 
|
help tracking down disk i/o
I have an mdadm raid 6 running with 8x 1tb drives. I am using a script that checks diff /proc/diskstats every 30 minutes to spin down the disks with hdparm -y (hdparm -s doesnt work on my system). last week, this used to work. i had 6x1tb drives in raid 6 with an ext3 fs.
installed 2 new disks, grew the array. fskc, no errors. resize2fs. fsck again, no errors. tune2fs to add extends, etc. (convert to ext4). fsck again, with -D option to optimize directory.
After doing the above, i now have constant disk ativity. it is not a lot, very little actually, but it is some. now the disks never spin down. this occurs always, after fresh reboot, for days now.
first, i read that when formatting an ext4 partition, it is just a quick format and lazyext4init will format in the background for the next few days or so. I've checked, i have no lazy process, nor did i actually format in the first place, just added extends and dir optimization.
second, ive tried to monitor whats going on. heres the results:
Code:
Every 1.0s: iostat sda sdb sdc sdd sdg sdh sdi sdj Wed Apr 2 12:23:47 2014
Linux 3.11.0-15-generic (fileserver) 04/02/2014 _x86_64_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
29.56 0.02 58.41 0.25 0.00 11.77
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 4.33 16.75 12.98 1214691 940830
sdb 4.36 16.92 12.98 1226747 940814
sdd 4.32 16.71 12.98 1211591 940810
sdc 4.34 16.80 12.98 1217915 940810
sdg 4.25 16.76 12.98 1215411 940814
sdh 4.27 16.89 12.98 1224535 940762
sdi 6.27 24.76 12.98 1794737 940814
sdj 4.30 16.94 12.98 1228172 940862
watching this, i see that all disks have a constant write to it, but again, very small. however, most disks the read does not change at all. the exception is one disk, sdi. every update, it reads more, while the others dont. this was a disk that was already previously in the 6x1tb array.
next, i set up inotifywait and recursively monitored my entire md array mount point. this showed absolutely no file/dir read/write/access, yet iostat keeps telling me there IS writes. this leads me to belive the access is not on the fs level, but more on the md level.
iotop shows nothing useful, a few jbd2 updates here and there for other non-md disks. i have tried mounting the md array with noatime, not that i thought it would make a difference.
Code:
administrator@fileserver:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid6 sdi1[0] sdg1[7] sdh1[6] sdd1[5] sdc1[4] sdb1[3] sda1[2] sdj1[1]
5859354624 blocks level 6, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]
bitmap: 0/8 pages [0KB], 65536KB chunk
unused devices: <none>
Code:
administrator@fileserver:~$ sudo mdadm --detail /dev/md2
/dev/md2:
Version : 0.90
Creation Time : Thu Feb 3 22:29:34 2011
Raid Level : raid6
Array Size : 5859354624 (5587.92 GiB 5999.98 GB)
Used Dev Size : 976559104 (931.32 GiB 1000.00 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Apr 2 12:28:48 2014
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : ae81aabc:ffec78b1:af80d5a0:0102b29f
Events : 0.6157437
Number Major Minor RaidDevice State
0 8 129 0 active sync /dev/sdi1
1 8 145 1 active sync /dev/sdj1
2 8 1 2 active sync /dev/sda1
3 8 17 3 active sync /dev/sdb1
4 8 33 4 active sync /dev/sdc1
5 8 49 5 active sync /dev/sdd1
6 8 113 6 active sync /dev/sdh1
7 8 97 7 active sync /dev/sdg1
Code:
administrator@fileserver:~$ sudo ps aux | grep md
root 34 0.0 0.0 0 0 ? S< Apr01 0:00 [md]
root 40 0.0 0.0 0 0 ? SN Apr01 0:00 [ksmd]
root 1205 0.0 0.0 0 0 ? S< Apr01 0:00 [krfcommd]
root 1941 0.0 0.0 0 0 ? S Apr01 0:27 [md2_raid6]
root 1980 0.0 0.0 0 0 ? S Apr01 0:00 [jbd2/md2-8]
1000 25144 0.0 0.0 13600 896 pts/4 S+ 12:30 0:00 grep --color=auto md
md2_raid6 and jbd2/md2-8 ARE active processes.... but no cpu activity and no listing in iotop.
where else to look for why there is constant disk activity?
thanks
mike |
|
|
|
|
04-02-2014, 02:04 PM
|
#2
|
|
Senior Member
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,232
|
You see the I/O activity in iostat on the individual disks. Do you also see it on md2? If not, it is some internal disk or md activity. If so, then it is from a layer above the block devices.
|
|
|
|
|
04-02-2014, 07:17 PM
|
#3
|
|
LQ Veteran
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,287
|
Have a quick check with iotop - it's what you'd expect it to be from the name.
If nothing shows up and you get sick of looking at that, try collectl - it collects the stats for you so you can post process it. Pidstat (from sysstat) only tracks I/O for active processes (last I looked at it anyway).
|
|
|
|
|
04-02-2014, 09:31 PM
|
#4
|
|
Member
Registered: Jun 2008
Posts: 122
Original Poster
Rep:
|
iotop does not show anything accessing the array. there are a few processes using disk io, but they wouldnt be using the array (jbd2/sde1 <--os drive, virtualbox <--separate storage drive)
Code:
1987 be/3 root 0.00 B/s 0.00 B/s 0.00 % 3.21 % [jbd2/sdf3-8]
334 be/3 root 0.00 B/s 0.00 B/s 0.00 % 1.24 % [jbd2/sde1-8]
3674 be/4 administ 3.85 K/s 7.70 K/s 0.00 % 0.01 % VirtualBox --commen~-startvm-errormsgbox
12481 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/u8:0]
collectl doesn't tell me anything that i dont already know. i'm not familiar with it, but after reading the man page, this is what ive got
Code:
# DISK STATISTICS (/sec)
# <---------reads---------><---------writes---------><--------averages--------> Pct
#Name KBytes Merged IOs Size KBytes Merged IOs Size RWSize QLen Wait SvcTim Util
sda 0 0 0 0 26 0 8 3 3 0 4 4 3
sdb 0 0 0 0 26 0 8 3 3 0 6 6 5
sdf 0 0 0 0 0 0 0 0 0 0 0 0 0
sdd 0 0 0 0 26 0 8 3 3 0 6 6 4
sdc 0 0 0 0 26 0 8 3 3 0 7 7 5
sdg 0 0 0 0 26 0 8 3 3 0 6 6 5
sdh 0 0 0 0 26 0 8 3 3 0 13 13 10
sde 0 0 0 0 0 0 0 0 0 0 0 0 0
sdi 16 0 4 4 26 0 8 3 3 0 48 48 57
sdj 0 0 0 0 26 0 8 3 3 0 10 10 7
i've unmounted the filesystem, yet the writes continue on all disks (and read from one)
Code:
administrator@fileserver:~$ mount
/dev/sde1 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
/dev/sdf3 on /mnt/storage type ext4 (rw)
administrator@fileserver:~$
Code:
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 4.23 11.64 12.98 1223239 1364653
sdb 4.25 11.75 12.98 1235407 1364637
sdd 4.22 11.61 12.98 1220163 1364633
sdc 4.24 11.67 12.98 1226723 1364633
sdg 4.17 11.65 12.98 1224395 1364645
sdh 4.19 11.73 12.98 1233475 1364585
sdi 6.18 19.64 12.98 2064693 1364642
sdj 4.21 11.77 12.98 1237260 1364693
so this is clearly not on the fs level.
***to note: md status says 'active' not 'clean'. before, when the disks spun down like they should, this would say 'clean'. i've got a feeling this is something mdadm is doing but i cant find any documentation on it or anything like it searching google. but, the mdadm process doesnt seem to have any i/o, or the tools just arent seeing it.
Code:
administrator@fileserver:~$ pidstat -d -p 1941 1 10
Linux 3.11.0-15-generic (fileserver) 04/02/2014 _x86_64_ (2 CPU)
09:38:28 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
09:38:29 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:30 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:31 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:32 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:33 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:34 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:35 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:36 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:37 PM 1941 0.00 0.00 0.00 md2_raid6
09:38:38 PM 1941 0.00 0.00 0.00 md2_raid6
Average: 1941 0.00 0.00 0.00 md2_raid6
administrator@fileserver:~$ pidstat -d -p 34 1 10
Linux 3.11.0-15-generic (fileserver) 04/02/2014 _x86_64_ (2 CPU)
09:38:44 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
09:38:45 PM 34 0.00 0.00 0.00 md
09:38:46 PM 34 0.00 0.00 0.00 md
09:38:47 PM 34 0.00 0.00 0.00 md
09:38:48 PM 34 0.00 0.00 0.00 md
09:38:49 PM 34 0.00 0.00 0.00 md
09:38:50 PM 34 0.00 0.00 0.00 md
09:38:51 PM 34 0.00 0.00 0.00 md
09:38:52 PM 34 0.00 0.00 0.00 md
09:38:53 PM 34 0.00 0.00 0.00 md
09:38:54 PM 34 0.00 0.00 0.00 md
Average: 34 0.00 0.00 0.00 md
Last edited by pepsimachine15; 04-02-2014 at 09:40 PM.
|
|
|
|
|
04-03-2014, 10:01 AM
|
#5
|
|
Senior Member
Registered: Feb 2011
Location: Massachusetts, USA
Distribution: Fedora
Posts: 4,232
|
To check I/O to the array:
To check for md doing something to the array:
Code:
cat /sys/block/md2/md/sync_action
|
|
|
|
|
04-03-2014, 07:25 PM
|
#6
|
|
Member
Registered: Jun 2008
Posts: 122
Original Poster
Rep:
|
Code:
administrator@fileserver:~$ iostat 1 | grep md2
md2 2.00 85.50 0.00 15690816 308
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
md2 0.00 0.00 0.00 0 0
^C
md2 doesnt look busy. the first line is totals since the device has been online, correct? i let it run for about 10 seconds, should have been more than enough to detect any activity (considering i can see the hd activity led flashing on the machine every second... blink blink blink - pause - repeat)
Code:
administrator@fileserver:~$ cat /sys/block/md2/md/sync_action
idle
md2 *says* its idle... but... its not!
Code:
administrator@fileserver:~$ sudo mdadm --stop /dev/md2
mdadm: stopped /dev/md2
administrator@fileserver:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: <none>
Code:
administrator@fileserver:~$ iostat 1 sda sdb sdc sdd sdg sdh sdi sdj
Linux 3.11.0-15-generic (fileserver) 04/03/2014 _x86_64_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
18.48 0.01 62.22 0.30 0.00 18.98
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 4.21 10.61 12.97 1953779 2388199
sdb 4.24 10.73 12.97 1975763 2388183
sdd 4.20 10.59 12.97 1948775 2388179
sdc 4.22 10.65 12.97 1959667 2388179
sdg 4.16 10.63 12.97 1956484 2388223
sdh 4.17 10.72 12.97 1973647 2388131
sdi 6.47 20.87 15.22 3840771 2801771
sdj 4.20 10.74 12.97 1977386 2388271
avg-cpu: %user %nice %system %iowait %steal %idle
12.56 0.00 73.87 0.00 0.00 13.57
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 0.00 0.00 0.00 0 0
sdd 0.00 0.00 0.00 0 0
sdc 0.00 0.00 0.00 0 0
sdg 0.00 0.00 0.00 0 0
sdh 0.00 0.00 0.00 0 0
sdi 0.00 0.00 0.00 0 0
sdj 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
11.44 0.00 67.66 0.00 0.00 20.90
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.00 0.00 0.00 0 0
sdb 0.00 0.00 0.00 0 0
sdd 0.00 0.00 0.00 0 0
sdc 0.00 0.00 0.00 0 0
sdg 0.00 0.00 0.00 0 0
sdh 0.00 0.00 0.00 0 0
sdi 0.00 0.00 0.00 0 0
sdj 0.00 0.00 0.00 0 0
it IS mdadm's fault... but it thinks that it is idle, and its clearly not. |
|
|
|
|
04-03-2014, 08:38 PM
|
#7
|
|
Moderator
Registered: Mar 2008
Posts: 22,186
|
Might depend on the disk. In some odd cases a drive cache causes some sort of rubber band sort of issue. If you have drive cache enabled then maybe disable it. Usually they are enabled.
Since you say new disks, I'd be considering that as a factor maybe?
|
|
|
|
|
04-03-2014, 09:00 PM
|
#8
|
|
Member
Registered: Jun 2008
Posts: 122
Original Poster
Rep:
|
i couldnt see the cache having anything to do with the writes to *all* disks, maybe one. either way, the cache should be cleared after power off, which i have done.
the new disks could be a factor. the only things that have changed since old 6x1tb array and new 8x1tb array with problems are:
1. grew array by 2 new disks (not actually *new* but spares i had)
2. expand filesystem
3. converted fs to ext4 from ext3
4. directory optimization for ext4 fs
5. added write intent bitmap on array
i ran smart short and extended tests last night on the only drive in the array doing the reads, passed both. i'll pick another to do tonight, they take long for extended and i'd prefer to do 1 at a time in an array.
|
|
|
|
|
04-04-2014, 03:28 PM
|
#9
|
|
Member
Registered: Jun 2008
Posts: 122
Original Poster
Rep:
|
smart tests on one of the two new drives added passed.
just for sh*ts n giggles, i took the 1 drive that has all the reads coming from it, sdi, and set it faulty/removed it from mdadm. at this point, sdj starts doing the reading, and the remaining 7 drives still have writes every second.
re-added sdi, now it is back to reading from sdi and writing to all 8 drives.
|
|
|
|
|
04-08-2014, 10:58 PM
|
#10
|
|
Member
Registered: Jun 2008
Posts: 122
Original Poster
Rep:
|
just an update - this has not been solved yet. well... not really....
had a power outage last night. batteries in the ups havent been changed in about 2 years, so runtime was under 5 minutes. this confused apcupsd and it did not shut down the machine in time before a hard off.
long story short, the machine came back online and mdadm decided to do a full resync of the array.
all the writes to the drives have now stopped. every drive spins down as it used to do, except for one. that one that does not spin down is the same drive that has been doing all the 'reading' the past two weeks. it is still doing the 'reading', just no writes to any disks.
|
|
|
|
All times are GMT -5. The time now is 02:03 PM.
|
|
|
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
|
|
 Latest Threads
LQ News
|
|
