When does the checking of mdadm happen?

bloupbloup · 11-05-2017, 02:58 AM

Without any reasons, all my arrays started to get checked. It was painful because it was during a backup and I guess it explained why it was delayed. Is the checking process scheduled somewhere? Can I predict when it will happen next time?
Does it happen after a reboot? Is there any logic behind it?

I have seen the following:

Code:

[Sun Nov  5 00:59:53 2017] md: data-check of RAID array md125
[Sun Nov  5 00:59:59 2017] md: delaying data-check of md126 until md125 has finished (they share one or more physical units)
[Sun Nov  5 01:00:05 2017] md: delaying data-check of md127 until md125 has finished (they share one or more physical units)
[Sun Nov  5 01:00:31 2017] md: md125: data-check done.
[Sun Nov  5 01:00:31 2017] md: delaying data-check of md127 until md126 has finished (they share one or more physical units)
[Sun Nov  5 01:00:31 2017] md: delaying data-check of md126 until md127 has finished (they share one or more physical units)
[Sun Nov  5 01:00:31 2017] md: data-check of RAID array md127
[Sun Nov  5 04:42:07 2017] md: md127: data-check done.
[Sun Nov  5 04:42:07 2017] md: data-check of RAID array md126
[Sun Nov  5 04:42:12 2017] md: md126: data-check done.

Code:

cat /proc/mdstat
Personalities : [raid1] 
md125 : active raid1 sdb1[1] sda1[0]
      4096000 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md126 : active raid1 sdb2[1] sda2[0]
      512960 blocks super 1.0 [2/2] [UU]
      	resync=DELAYED
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sdb3[1] sda3[0]
      972018688 blocks super 1.2 [2/2] [UU]
      [==================>..]  check = 94.2% (915822080/972018688) finish=27.4min speed=34068K/sec
      bitmap: 4/8 pages [16KB], 65536KB chunk

unused devices: <none>

deep27ak · 11-05-2017, 03:21 AM

Your disks are syncing which happens only if a new disk was added to your existing raid array.
In general once the disks are synced completely then the sync happens backend without affecting any process writing to that part of disk.

Check you syslog to find more details if someone performed a disk replacement?

Shadow_7 · 11-05-2017, 03:26 AM

Are there cron jobs scheduled for it?

Otherwise check the daemons that might launch it. With strings and grep on the binaries listed in your ps output. And the related .conf and sources for those processes. There likely is logic behind it, although maybe not predictable logic.

bloupbloup · 11-05-2017, 03:59 AM

Quote:

Originally Posted by Shadow_7

Are there cron jobs scheduled for it?

Otherwise check the daemons that might launch it. With strings and grep on the binaries listed in your ps output. And the related .conf and sources for those processes. There likely is logic behind it, although maybe not predictable logic.

I have found the root cause of the issue in a cron job:

/etc/cron.d/raid-check cron job running /usr/sbin/raid-check at 1am every sunday.

this is on my centos 7.

thank you.

voleg · 11-08-2017, 01:46 PM

Then check /etc/sysconfig/raid-check, they should come together.

frater · 08-02-2020, 10:21 AM

I am monitoring /proc/mdstat and /sys/block/md1/md/sync_speed with Zabbix, so I know when my RAIDs are syncing.
This Sunday morning I saw 2 of my servers having a high /sys/block/md1/md/sync_speed and I thought they were both syncing.

It started at the exact same time which could indicate a power cut.
This happened before, but after I issued a "uptime" I knew it had to be something else.
Thanks to this thread I was relieved to find out it was "checking" the RAID and not "syncing" it.

I now need to modify my triggers a bit, so it can distinguish between syncing and checking.

I looked for a cron job on mine and found /etc/cron.d/mdadm
This is on my Ubuntu 18.04

Code:

# cat /etc/cron.d/mdadm
#
# cron.d/mdadm -- schedules periodic redundancy checks of MD devices
#
# Copyright © martin f. krafft <madduck@madduck.net>
# distributed under the terms of the Artistic Licence 2.0
#

# By default, run at 00:57 on every Sunday, but do nothing unless the day of
# the month is less than or equal to 7. Thus, only run on the first Sunday of
# each month. crontab(5) sucks, unfortunately, in this regard; therefore this
# hack (see #380425).
57 0 * * 0 root if [ -x /usr/share/mdadm/checkarray ] && [ $(date +\%d) -le 7 ]; then /usr/share/mdadm/checkarray --cron --all --idle --quiet; fi