Drive stuck, how to I un-stuck it.

MikeyCarter · 08-05-2009, 07:24 AM

I've had this happen twice now in the last 6 months. I know a reboot will fix it. Just trying to figure out how to fix it without the reboot.

I was doing a very large rsync when all of a second it froze.

Code:

fuser -m /dev/Surplus/VirtualStorage 
/dev/Surplus/VirtualStorage: 16312c

ps aux | grep 16312
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     16312  1.0  0.0  90160   720 pts/8    D    Aug04   9:02 rsync -a --progress firefly-crash.img firefly-php.img firefly-solaris.img firefly-ubuntu.img /mnt/temp/

ps -eflH | grep 16312
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
1 D root     16312     1  1  80   0 - 22540 sync_p Aug04 pts/8    00:09:02   rsync -a --progress firefly-crash.img firefly-php.img firefly-solaris.img firefly-ubuntu.img /mnt/temp/

grep sync_p /proc/*/wchan
/proc/16312/wchan:sync_page


 lsof /dev/Surplus/VirtualStorage 
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
rsync    16312 root  cwd    DIR  253,7     4096    2 /mnt/temp

Currently any command executed on this directory /mnt/temp is being held. (sync, updatedb, ls, anything) I've tried killing the rsync but no go.

Any ideas how to unlock it? This file system is ext4, and the previous lock was on a xfs file system, if it helps.

kilgoretrout · 08-05-2009, 11:40 AM

If you haven't tried killing the processes accessing the device with fuser, I'd give that a shot:

# fuser -km /dev/Surplus/VirtualStorage

That should kill all processes accessing the device.

MikeyCarter · 08-05-2009, 11:51 AM

fuser -km /dev/Surplus/VirtualStorage doesn't work. Process is still there. Found the main process which I think is holding the lock:

F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
1 D root 15222 2 0 75 -5 - 0 jbd2_j Aug04 ? 00:00:19 [kjournald2]

Can't kill it either. (not that I should kill kjournald2, but being a new partition I knew I could always re-format if I corrupted it)

GrapefruiTgirl · 08-05-2009, 11:57 AM

I don't know much about fuse/fuser so this might be totally worthless:

If the problem is arising from hardware, i.e. a real physical drive is 'frozen' or confused, the 'hdparm' and 'sdparm' tools can be used to force a reset of the drive. I have never had it fail, though depending on how confused was the drive, it sometimes took several seconds up to as long as 20-30 seconds to un-freeze the drive.

I believe the hdparm option is -W as in `hdparm -W /dev/hda` but verify this on the man page.

Sorry if I'm barking up the wrong tree here,

Sasha