[SOLVED] uniterruplible sleep mode for an rm process

nass · 07-25-2011, 08:18 AM

hello everyone,
I haven't done this before and it seems my reasoning fails me somewhere.

The problem is that I am trying to remove some snapshots on the LVM disk of a backup server.
At the moment I'm doing some snapshot testing with rsnapshot. now I want to delete all the test snapshots and set my cron jobs to take snapshots automatically.
Problem is rm -rf /mnt/hd/snaps runs indefinitely.
looking closer:

Quote:

root@samothraki:/mnt/hd/snaps# ps aux | grep rm
root 2219 0.4 0.0 4488 568 pts/1 D+ 15:50 0:00 rm -rf 4hrly.1 dly.0

this D (uninterruptible sleep state) as I read on the internet can only be woken if the block device becomes available again. (say if I unplug nfs shares that could happen, but I don't try to remove files from an nfs share)

Now there is a number of things that could have gone wrong here but none that I can think of. so here are the facts:

-this backup server is running slack64 13.37 , slightly modified generic kernel.
-I have a few disks in it. The LVM disks where I keep my snapshots, is made up of 2x2TB (raid 1 attached to 3ware raid card), 2x1.5TB (soft raid1) and 2x750GB (again soft raid 1). Altogether I have (and NEED to have) 2+1.5+0.75=4.25TB of backup space in order to get backups from another server.
(note as you can see this lvm disk is somewhat not standard - could there be a problem here?)
- I have written a little script (see below) to create and mount some necessary nfs directories in order to mount whatever I want to backup from the file server.
- I call the script from rsnapshot.conf. On exit I umount all these shares, just for the shake of cleanness.

The lvm disk info is:

Code:

/dev/lvm_disk/lvm_node  /mnt/hd         xfs     defaults        1       2

I am unsure as to what prevents the rm command from executing. To me the likelyhood is something is not right with the lvm (some information that the rm command should receive it does not receive for some reason ?)

ANY pointers and guidance will be greatly appreciated

Thank you in advance
Nass

EDIT: I managed to narrow down the problematic folder. These are the mountpoint directories. Let me explain:

I create the necessary mountpoints in the backup server under /mnt/nfs_asty/.
once rsnapshot commenced the backup operation, it copies the full path of these locations under each snapshot dir.

So a snapshot will be located under
/mnt/hd/snaps/<snapshot number>/<data host>/mnt/nfs_asty/

it looks like:

Code:

root@samothraki:/mnt/hd/snaps/4hrly.1/astypalaia/mnt/nfs_asty# ls -l
total 200
drwxrwxrwx+   3  507 root   55 Feb 28 12:43 asil/
drwxrwxrwt+  25 root root 4096 Jul 25 15:40 backups/
drwxrwxrwx+   2  528 root    6 Feb 16 18:13 dbad/
drwxrwxrwx+   2  536 root    6 Jun 15 12:18 dbra/
drwxrwxrwx+  12 root root 4096 Jan 27 13:57 drivers/
drwxrwxrwx+   2  506 root    6 Feb 14 17:53 dsot/
drwxrwxrwx+   2 root root    6 Feb 14 22:05 email_st/
drwxrwxrwx+   5 root root 4096 Jul 14 12:25 emails/
drwxrwxrwx+   2  532 root    6 Feb 26 17:06 gbur/
drwxrwxrwx+   2  515 root    6 Feb 14 17:58 gcho/
drwxrwxrwx+   9  530 root 4096 Apr  7 11:42 grammateia/
drwxrwxrwx+   3  505 root   20 Feb 28 13:21 isak/
drwxrwxrwx+   2 root root    6 Feb 15 15:59 iso/
drwxrwxrwx+   2  534 root    6 Mar  2 21:41 ithaki/
drwxrwxrwx+   2  522 root    6 Feb 14 18:35 itsi/
drwxrwxrwx+   2  519 root    6 Feb 14 18:32 kant/
drwxrwxrwx+   2  518 root    6 Feb 14 18:31 kstr/
drwxrwxrwx+   3  513 root   20 Feb 28 12:29 kzac/
drwxrwxrwx+   2  533 root    6 Mar  2 21:40 leo/
drwxrwxrwx+   2  523 root    6 Feb 14 18:36 lhav/
drwxrwxrwx+   2  525 root    6 Feb 14 18:39 lkot/
drwxrwxrwx+   4  529 root   34 Feb 28 14:19 llyt/
drwxrwxrwx+   5 root root   62 Jul 23 03:02 local_backups/
drwxrwxrwx+ 134 root root 8192 Jul 25 16:29 meletes/
drwxrwxrwx+   2  509 root    6 Feb 14 18:38 mkaf/
drwxrwxrwx+   3  510 root   20 Feb 26 18:26 msot/
drwxrwxrwx+   3  500 root   37 Feb 18 19:26 nass/
drwxrwxrwx+   3  508 root   20 Feb 28 10:51 nsil/
drwxrwxrwx+  16 root root 4096 Feb 27 19:56 old_backups/
drwxrwxrwx+   6 root root  103 Jul 29  2010 old_scans/
drwxrwxrwx+   2  512 root    6 Feb 14 17:54 pfil/
drwxrwxrwx+   3  524 root   20 Feb 28 14:08 pkat/
drwxrwxrwx+   2  527 root    6 Feb 16 11:18 prup/
drwxrwxrwx+   2 root root 8192 Jul 21 17:12 scans/
drwxrwxrwx+   2  521 root    6 Feb 14 18:34 span/
drwxrwxrwx+   3  531 root   18 Feb 24 20:14 tameio/
drwxrwxrwx+   3  517 root   20 Feb 28 10:42 tlam/
drwxrwxrwx+   4  514 root   31 Feb 28 10:40 vner/
drwxrwxrwx+   3  516 root   20 Feb 28 13:40 ztro/

if I try to delete any of these folders, rm goes to D state.
I can empty the contents of each of these folders though without a problem.
Any clue as to what could be the problem?

Now, if I try to delete any of the folders, rm goes to D state.
but i CAN empty the contents of these folder without a problem.
Any clue as to what could be the problem?

nass · 07-26-2011, 06:18 AM

OK please disregard completely this thread.
The problem was not with rm going into D uninterruptible state.
I managed to complete the removal of the rnsapshot directory. Manually in the beginning, i realised the deletions (unlinks) would go forth, in the end I left it alone (rm -rf /mnt/hd/snaps/) and the process managed to delete 1.7TB of data in approx 8hours....

looking around I found out that xfs is (can be?) terrible at unlinking directories especially whene there are many and with deep file structures. see here http://www.t2-project.org/zine/1/

In the end I reformatted to jfs, seems quite alright now. Only big drawback is its inability to shrink. But that's not important for an exclusive backup disk.

I'm going to mark the thread as solved even though that won't be a solution to the initial problem

A word of advice if I may , DON'T use xfs with deep folder structures. This is especially true with rsnapshot :P

syg00 · 07-26-2011, 06:32 AM

Thanks for the follow-up. I was going to query xfs, but as I don't use it I didn't want to go pointing fingers ...

Richard Cranium · 07-26-2011, 09:13 AM

http://seife.kernalert.de/blog/2009/...k-performance/