resize2fs beeing stuck cpu at 0%

sierramike · 08-07-2011, 09:02 AM

Hi,

I've got a huge problem. I needed to shrink a filesystem so I could take a drive out of a RAID5 array.

I unmounted the fs, then ran e2fsck on it and it found 5.9% non contiguous files but no errors.

Then I ran resize2fs -f /dev/whatever 1300G (initial size was 2500G, used space was at this time only 1136G). It started operating and after some hours I received a lot of messages on the output as mdadm started resync in the background. Now the resync is done but nothing moves.

Here is the output I received in the terminal window where resize2fs was ran :

Code:

computer:/# e2fsck /dev/stold/storold
e2fsck 1.41.3 (12-Oct-2008)
/dev/st/stor a été monté 68 fois sans avoir été vérifié, vérification forcée.
Passe*1*: vérification des i-noeuds, des blocs et des tailles
Passe 2*: vérification de la structure des répertoires
Passe*3*: vérification de la connectivité des répertoires
Passe 4*: vérification des compteurs de référence
Passe 5*: vérification de l'information du sommaire de groupe
/dev/st/stor*: 107605/152592384 fichiers (5.9% non contigus), 312781819/610368512 blocs
computer:/# resize2fs -f /dev/st/stor 1300G
resize2fs 1.41.3 (12-Oct-2008)
Resizing the filesystem on /dev/st/stor to 340787200 (4k) blocks.

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.989687] Process md0_raid5 (pid: 872, ti=dd810000 task=df44f920 task.ti=dd810000)

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.989715] Stack: c13b2700 c60c2ce0 00000001 00000000 00000000 00000000 00000000 dd92b77c

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.989778]        c60c2ce0 dd92b7c8 e0938abb 2e3c3278 00000000 00000000 df4c3720 00000018

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.989840]        00000000 00000000 00000004 00000000 d5768aa0 00000246 de186a60 00000000

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.989902] Call Trace:

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.989942]  [<e0938abb>] raid5_run_ops+0x1ec/0x482 [raid456]

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.989176] Oops: 0000 [#1] SMP

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.989990]  [<c0158d0d>] mempool_free+0x63/0x67

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990038]  [<e093987b>] handle_stripe5+0xb2a/0xb3b [raid456]

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990092]  [<e093b79c>] handle_stripe+0xbb7/0xc07 [raid456]

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990132]  [<c0134456>] hrtimer_start+0xf7/0x110

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990168]  [<c011d43b>] hrtick_set+0x8f/0xd8

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990204]  [<c011a6bb>] __wake_up+0x29/0x39

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990250]  [<e0902b51>] md_wakeup_thread+0x1e/0x20 [md_mod]

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990325]  [<e0936906>] release_stripe+0x21/0x2e [raid456]

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990363]  [<e093bb63>] raid5d+0x377/0x385 [raid456]

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990398]  [<c02b8bd1>] schedule_timeout+0x13/0x86

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990448]  [<e09093c7>] md_thread+0xb7/0xcd [md_mod]

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990490]  [<c01319d4>] autoremove_wake_function+0x0/0x2d

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990532]  [<e0909310>] md_thread+0x0/0xcd [md_mod]

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990572]  [<c0131913>] kthread+0x38/0x5d

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990599]  [<c01318db>] kthread+0x0/0x5d

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990627]  [<c01044f7>] kernel_thread_helper+0x7/0x10

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990667]  =======================

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990688] Code: 44 24 04 0f b7 40 1a c7 44 24 0c 00 00 00 00 6b c8 0c 89 44 24 14 89 4c 24 10 eb 7c 8b 44 24 04 31 c9 8b 54 24 10 03 50 30 85 f6 <8b> 7a 04 79 08 89 f1 f7 d9 01 ce 29 cf 85 ff 7e 51 8d 04 3e 89

Message from syslogd@computer at Aug  7 00:39:34 ...
 kernel:[29153.990919] EIP: [<e09378ce>] async_copy_data+0x6a/0xf9 [raid456] SS:ESP 0068:dd811de0

Here is what I did in another terminal :

Code:

computer:/# ps aux | grep resize2fs
root      3845 11.5 57.8 301236 298844 pts/0   D+   Aug06 137:37 resize2fs -f /dev/st/stor 1300G
root     24878  0.0  0.1   3144   776 pts/1    S+   15:45   0:00 grep resize2fs
computer:/# top -p 3845
top - 15:59:06 up 23:25,  2 users,  load average: 4.00, 4.00, 4.00
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 25.0%id, 75.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    516256k total,   474868k used,    41388k free,    53336k buffers
Swap:  3927884k total,      704k used,  3927180k free,    70268k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3845 root      20   0  294m 291m  868 D  0.0 57.9 137:37.98 resize2fs

I understand that it doesn't do nothing anymore, as it doesn't consume any CPU cycles anymore.

What can I do now ? It's running near 24 hours now !

How can I try to escape from this situation without loosing all data ?

Thanks so much by advance ...

syg00 · 08-07-2011, 01:15 PM

Doesn't look good.
Why did you force the resize ?.
What makes you think the re-sync finished ?. That looks like an oops - probably in mdadm somewhere. Your resize is in uninterruptible sleep - it won't go anywhere until the operation it is waiting on responds. That's probably an I/O, and mdadm is in control of the backing block devices.

I guess asking if you took a backup before you started is just me being hopeful.

sierramike · 08-07-2011, 01:24 PM

Well, no backup, this is an enourmous place where I store all my multimedia files, I don't have anything big enough to store such an amount of data ...

But, I crossed my fingers and went with a reboot. Actually the filesystem did mount without errors after reboot, I just unmounted it and started a e2fsck -f on it in order to see if there are any errors (without the -f flag it told me the volume is clean ...).

I saw it didn't do the resize, so maybe it stopped its operation in a clean state ...

I knew the resync did finish because I did a "watch -n1 cat /proc/mdstat" and kept looking until all resync finished, then waited for hours before even posting here ...

Actually waiting for e2fsck to finish ... I'll write here if it's ok or not ...

sierramike · 08-07-2011, 03:44 PM

Apparently, all is ok, no error reported by e2fsck ...

So I just have to go further !

resize2fs seems to be a good tool thought, it didn't break anything !