Problems with an USB hard disk (ext3)

cabrilo · 02-17-2011, 02:41 PM

A couple of days ago I attached an USB hard disk (it's SATA) to a remote server (fedora 13) I formatted it it ext3 before, and everything seemed to work for a couple of days. This morning, when I cd'd to it's directory and tried to ls, I got the following error:

ls: reading directory .: Input/output error

So I checked the logs. I initially attached it as sdb1, though I used /dev/disk/by-id/whatever. This is that first attachment:

Code:

Feb 15 10:18:04 linux kernel: sdb: sdb1
Feb 15 10:18:04 linux kernel: sd 2:0:0:0: [sdb] Assuming drive cache: write through
Feb 15 10:18:04 linux kernel: sd 2:0:0:0: [sdb] Attached SCSI disk
Feb 15 10:20:45 linux kernel: EXT3-fs (sdb1): using internal journal
Feb 15 10:20:45 linux kernel: EXT3-fs (sdb1): mounted filesystem with ordered data mode

This is when I noticed the problem.

Code:

Feb 17 09:01:26 linux kernel: EXT3-fs error (device sdb1): ext3_find_entry: reading directory #2 offset 0 (a bunch of errors likes this)
Feb 17 09:58:07 linux kernel: EXT3-fs error (device sdb1): ext3_get_inode_loc: unable to read inode block - inode=2, block=1027
Feb 17 09:58:07 linux kernel: EXT3-fs (sdb1): error in ext3_reserve_inode_write: IO failure
Feb 17 09:58:13 linux kernel: journal_bmap: journal block not found at offset 9444 on sdb1
Feb 17 09:58:13 linux kernel: Aborting journal on device sdb1.
Feb 17 09:59:24 linux kernel: EXT3-fs error (device sdb1): ext3_find_entry: reading directory #2 offset 0
Feb 17 10:02:01 linux kernel: EXT3-fs (sdb1): error: ext3_put_super: Couldn't clean up the journal
Feb 17 10:02:01 linux kernel: EXT3-fs (sdb1): error: remounting filesystem read-only

I did umount /dev/disk/by-id/diskid (where diskid is it's id).

Then I did mount /dev/disk/by-id/diskid and it mounted properly, but /dev/disk/by-id/diskid was now linked to sdc!

So, I grep'ed sdc in /var/log/messages, and I found these curious logs. Check the time. It happened at 23:15 on Feb 16, while the disk was mounted as /dev/sdb!

Code:

Feb 16 23:15:27 linux kernel: sd 3:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
Feb 16 23:15:27 linux kernel: sd 3:0:0:0: [sdc] Write Protect is off
Feb 16 23:15:27 linux kernel: sd 3:0:0:0: [sdc] Assuming drive cache: write through
Feb 16 23:15:27 linux kernel: sd 3:0:0:0: [sdc] Assuming drive cache: write through
Feb 16 23:15:27 linux kernel: sdc: sdc1
Feb 16 23:15:27 linux kernel: sd 3:0:0:0: [sdc] Assuming drive cache: write through
Feb 16 23:15:27 linux kernel: sd 3:0:0:0: [sdc] Attached SCSI disk

Any ideas what caused this? This is a remote server, so I didn't have a chance to see what's actually going on with the USB enclosure, if it's power on and such.

fsck.ext3 /dev/sdc1 was clean, and e2fsck -f /dev/sdc1 looks fine as well.

Any tips and ideas would be much appreciated,
Dejan

kbp · 02-17-2011, 06:07 PM

See if another device took /dev/sdb ..

syg00 · 02-18-2011, 12:41 AM

I'd be thinking a power drop - then the udev scripts kicked in again on an "add" event when the (physical) device re-appeared. Dodgy connector (or cleaners bumping it or using the power point ...) might look the same.
Check your environmental logs and security cameras.

cabrilo · 02-18-2011, 01:18 AM

Quote:

Originally Posted by kbp

See if another device took /dev/sdb ..

No, sdb is empty.

Quote:

Originally Posted by syg00

I'd be thinking a power drop - then the udev scripts kicked in again on an "add" event when the (physical) device re-appeared. Dodgy connector (or cleaners bumping it or using the power point ...) might look the same.
Check your environmental logs and security cameras.

I'm thinking the same thing, but no humans were involved in this.

The HDD is in an USB enclosure, directly connected to the server and powered off of the same grid as the server itself - so if there were a power failure, UPS would give me a notification of some sort.

Is it possible that the HDD just powered down with no particular reason? Is it known to happen?

Thanks

Raveolution · 02-18-2011, 01:23 AM

I am not sure why that would happen. You need physical access to the drive at the time that this occurs in order to solve this.

But in the meantime may I suggest adapting your system to mounting the drive by its (presumably) unchanging uuid?

You can find it by going to /dev/disk/by-uuid and seeing which uuid is linked to the current entry you know of in /dev/disk/by-id.

The UUID is the filename under /dev/disk/by-uuid and you'd put an entry for it in /etc/fstab.

cabrilo · 02-18-2011, 04:14 AM

Quote:

Originally Posted by Raveolution

But in the meantime may I suggest adapting your system to mounting the drive by its (presumably) unchanging uuid?
The UUID is the filename under /dev/disk/by-uuid and you'd put an entry for it in /etc/fstab.

Is uuid any better than id? Currently, I have it in fstab by /dev/disk/by-id/whatever. Would I gain anything with uuid?

Thanks

syg00 · 02-18-2011, 04:44 AM