Yesterday I had this scary experience. Let me tell what happened.

My server has 2 hard disks, sda and sdb. Both contain 5 partitions. Each partition is part of a RAID1 array. The boot partition contains all normal Linux partitions except /home. So /, /var, /usr /tmp are all on /dev/md0, which is built of /dev/sda1 and dev/sdb1.

To this server I have attached an external USB disk which I use for daily backups. (The weekly backups are performed on tape.) This external USB disk is recognized as /dev/sdc, and the partition /dev/sdc1 is mounted in /mnt/ext_daily. The mounting is manual.

No problem at all.

Yesterday we had to shutdown the server twice for maintenance of the server room UPS. We shut down all the systems, and plugged the server mains line into the grids and started up all servers. No problems. I did not do any checks because the server in question booted without problems and evertyhing was accessible.

Four hours later we followed the same procedure of shutting down and restarting to power the server room again from the UPS.

Server booted fine and this time I wanted to mount the external USB disk again. To my surprise the /dev/md0 was comprised of /dev/sdb1 and /dev/sdc1 (instead of /dev/sda1 and /dev/sdb1) and the external USB disk had become /dev/sda. Indeed, I had forgotten to unplug the USB disk during booting, not realizing that the device assignment could be different.

However, the array was functioning OK, and apparently it was assembled correctly using the UUID's and not the device names. I was in doubt whether I should reboot again now most users would have connected to the Samba shares again to correct this situation. I decided not to. And I mounted /dev/sda1 (the external USB disk) on /mnt/ext_daily.

After a few minutes, user complaints started to come in. No access to the Samba shares.

I noticed two things:
a. The root partition was 100% used 9.2 GB out of 9.2 GB.
b. The process /dev/md0 was using about 6% processor capacity, giving load figures of 2.0 - 2.8 in top. The hard disk light was continuously on.

(a) was perfectly possible. Logrotate was set up incorrectly on this machine and both system.log and daemon.log had grown to 3.1 GB each. The rest of the installation could account for another 3 GB, so 9.2 GB was viable.

(b) This I didn't understand, and I was scared that mdadm was trying to sync my external USB drive (now /dev/sda1). /proc/mdstat did not show anything of this syncing though.

First thing I did was to unmount /dev/sda1, which did not stop the CPU usage for /dev/mdadm.

When I tried to restart services this failed, but it was clear that this was a result of the 100% usage of root partition.

When I deleted daemon.log and system.log, du showed a disk usage of about 3 GB, as to be expected. However, df still showed a usage of 9.2 GB and 100%. This really convinced me that something terrible had happened to the file system, like mdadm overwriting the etx3 file system on /dev/md0 with data on /dev/sdb1 or so. fsck -n showed a clean drive (but still with 100% disk usage!). But then again, fsck.ext3 warns for unreliable results on mounted partitions.

I shut down the machine again and booted from a live USB disk. The USB stick was RAID capable. I re-assembled the /dev/md0 RAID partition, and to my surprise disk use was now 2.8GB, just what it should have been. fsck returned no errors again. The external USB disk looked sound as well.

So this time I disconnected the USB disk and rebooted. The RAID came up using sda and sdb, the external USB disk was sdc again, all problems solved.

So I did learn to either unplug the USB disk before booting, or configure udev correctly.

I am not sure if mdadm had tried to sync with the external USB disk. Frankly I don't think so. mdadm assembles based on UUID's, not on device names.

But what could have caused the 100% and 9.2 GB of disk usage while I had only 3 GB worth of files? WHy was this space not reclaimed? On this file system, I was using 13% of all available inodes.

jlinkels

Thanks for the warning, great story, i.e. well written.

Thanks for the warning, great story, i.e. well written.
Amazing your exprience....

0 members found this post helpful.