rsync doesn't seem to delete

cooks44 · 11-15-2007, 08:22 PM

Hi All,

Still learning and always looking for a source of new answers when I can't find one myself, so I thought I would try here.

I've got a script using rsync 2.6.4 to backup a large RAID array to a separate NAS box I have. All seemed to be working fine, however I've just checked the logs to find it isn't working... It seems that the --delete option isn't deleting and the NAS is now full.

The command I'm running is:

rsync -vrutW --delete /usr/local/graphicsraid/storage/ /mnt/mirror/

the NAS being mounted through samba (it doesn't support rsync natively, well, the current firmware it has doesn't anyway).

When I run the command with the --dry-run option, all looks fine - I can see the many files that it intends to delete. I remove the dry-run option, and rather than delete first it starts copying immediately. Of course it runs out of space with a "No space left on device (28)" error soon after.

I just can't think of a reason it is doing it like this...

Would anyone be able to shed some light on the situation? Thanks!

neek · 11-16-2007, 02:10 AM

Quote:

Originally Posted by cooks44

The command I'm running is:

rsync -vrutW --delete /usr/local/graphicsraid/storage/ /mnt/mirror/

You're copying a large amount of data from /usr/local/graphicsraid/storage to /mnt/mirror, asking rsync to delete files from /usr/local/graphicsraid/storage when it has finished, and /mnt/mirror is becoming full?

Sounds pretty normal to me. Are you sure /mnt/mirror has enough space for the files you're copying there? Compare 'du -ks /usr/local/graphicsraid/storage' against 'df -k /mnt/mirror' and confirm there is enough space on your NAS for all the data from your RAID.

Or maybe I misunderstand your question, or rsync, or both

Quigi · 11-16-2007, 01:48 PM

Neek, you may misunderstand rsync.

Quote:

Originally Posted by man rsync

--delete
This tells rsync to delete any files on the receiving side that
aren’t on the sending side.

So it should remove files from /mnt/mirror/ that don't (any longer) exist on /usr/local/graphicsraid/storage/. I'd say that's what Cooks44 intended, and verified in a dry run. And, as there's no --delete-after option, we get the default behavior:

Quote:

By default rsync does file deletions on the receiving side
before transferring files to try to ensure that there is suffi-
cient space on the receiving filesystem.

So maybe an I/O error prevents the deletion (which you could ignore with --ignore-errors). Or there may be a bug in rsync?

BTW, accessing a mounted (remote?) file system undoes the benefits of the ingenious rsync protocol, which can bring a file that is already at the recipient in sync by only transferring block changes, sometimes significantly reducing network bandwidth usage. But that may not be an issue for Cooks44.

Sorry, I don't have a clear route to troubleshooting. Maybe (A) try a small subtree, (B) use --ignore-errors there, (C) find a rsync dedicated mailing list, (D) look at the source code, (E) strace, (F) pull out your hair?

/Quigi

LnxAdm · 11-16-2007, 02:20 PM

My guess is that your rsync is dying before it even gets around to deleting the files.
Are you getting errors?

kav · 11-16-2007, 02:45 PM

Yeah it sounds like he needs to put --delete in there

Quigi · 11-17-2007, 08:18 AM

See the original post --
LnxAdm: Nope, it doesn't die. It skips deleting, and starts copying (which should happen after).
kav: Cooks44 did use --delete. I just had to quote the rsync man page to Neek, who suspected he misunderstood.

neek · 11-17-2007, 09:57 AM

Quote:

Originally Posted by Quigi

Neek, you may misunderstand rsync.

Indeed

I mis-read my version of the man page, which is possibly the Debian version on my Fedora Core 6 box and reads "--delete delete extraneous files from dest dirs", for some reason I confused this with the --remove-sent-files option on the version of rsync we use on Solaris :P Sorry. My man page does state "--delete-before receiver deletes before transfer (default)" and so with --delete it should be deleting before transfer, as the OP said --dry-run indicated.

Quote:

Originally Posted by Quigi

BTW, accessing a mounted (remote?) file system undoes the benefits of the ingenious rsync protocol, which can bring a file that is already at the recipient in sync by only transferring block changes, sometimes significantly reducing network bandwidth usage. But that may not be an issue for Cooks44.

He used -W, which my man page says "-W, --whole-file copy files whole (without rsync algorithm)" so good point, but Cook44 may already have taken it into account.

I like the strace suggestion; try to figure out when the unlink operation is attempted and what the return value is from the system call. I've tested rsync on a tmp source/dest dir, with something simple like "strace <original rsync command line>" but cannot see unlink being performed though the removed source file is removed from the destination dir, but I probably need the strace man page quoted at me :P Maybe someone can suggest a fix to this, probably obviously mistaken, strace usage.

cooks44 · 11-18-2007, 04:44 PM

Yes, the --delete option was set, has always been. I've tried "--delete-before", and a few other combinations to ensure it's using default behaviour.

I'm mirroring a 1TB RAID on my main graphics server to a 1TB JBOD on a NAS I have. It's all running locally on gigabit, so saving bandwidth through rsync algorithms isn't necessary - saving time on the job is. From using rsync in the past it is the perfect tool, so I'll stick with it and figure it out.

I have thrown an --ignore-errors into the scripts too, hoping that might show up an I/O error to no avail.

As listed in my OP, it's so strange that the behaviour I expect is happening in the dry run, but not when it's actually performed.

Either way, you have given me a few avenues to try - I'm leaning towards Samba not liking me (it never does...) so I'll see if I can trace it out and figure out why.

Thanks for your help!

Quigi · 11-21-2007, 11:34 AM

First, doggedly pursuing strace:

Quote:

Originally Posted by neek

I like the strace suggestion; try to figure out when the unlink operation is attempted and what the return value is from the system call. I've tested rsync on a tmp source/dest dir, with something simple like "strace <original rsync command line>" but cannot see unlink being performed though the removed source file is removed from the destination dir, but I probably need the strace man page quoted at me :P Maybe someone can suggest a fix to this, probably obviously mistaken, strace usage.

It needs the -f option to follow children. (Apparently rsync doesn't call fork but clone, which seems similar). Here is a tiny example rsync use.

Code:

$ mkdir orig mirror
$ touch orig/{a,b,c,d}
$ rsync -vrutW --delete orig/ mirror/
building file list ... done
./
a
b
c
d
                                                                                                                          
sent 242 bytes  received 114 bytes  712.00 bytes/sec
total size is 0  speedup is 0.00
$ rm orig/{a,c}
$ strace -f -o rsync.strace rsync -vrutW --delete orig/ mirror/
building file list ... done
deleting c
deleting a
./
 
sent 64 bytes  received 26 bytes  180.00 bytes/sec
total size is 0  speedup is 0.00
$

The deleting worked. Both orig and mirror are on a local file system, Samba is NOT involved. The resulting file rsync.strace has 392 lines. It shows that the process 26540 had a child 26541, which did the unlinking, and a grandchild 26542. Here's an excerpt:

Code:

26540 execve("/usr/bin/rsync", ["rsync", "-vrutW", "--delete", "orig/", "mirror/"], [/* 20 vars */]) = 0
...
26540 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7e2d708) = 26541
26541 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7e2d708) = 26542
...
26541 lstat64("c", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
26541 lstat64("d", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
26541 getdents64(0, /* 0 entries */, 4096) = 0
26541 close(0)                          = 0
26541 unlink("c")                       = 0
26541 select(4, [3], [1], NULL, {60, 0}) = 1 (out [1], left {60, 0})
26541 write(1, "\1\0\0lc", 5 <unfinished ...>
26540 <... select resumed> )            = 1 (in [5], left {59, 997000})
26540 read(5, "\1\0\0l", 4)             = 4
26540 select(6, [5], [], NULL, {60, 0}) = 1 (in [5], left {60, 0})
26540 read(5, "c", 1)                   = 1
26540 write(1, "deleting c\n", 11)      = 11
26540 select(6, [5], [], NULL, {60, 0} <unfinished ...>
26541 <... write resumed> )             = 5
26541 time(NULL)                        = 1195601625
26541 unlink("a")                       = 0
26541 select(4, [3], [1], NULL, {60, 0}) = 1 (out [1], left {60, 0})
26541 write(1, "\1\0\0la", 5 <unfinished ...>
26540 <... select resumed> )            = 1 (in [5], left {59, 999000})
26540 read(5, "\1\0\0l", 4)             = 4
26540 select(6, [5], [], NULL, {60, 0}) = 1 (in [5], left {60, 0})
26540 read(5, "a", 1)                   = 1
26540 write(1, "deleting a\n", 11)      = 11
26540 select(6, [5], [], NULL, {60, 0} <unfinished ...>
26541 <... write resumed> )             = 5
...

So much for strace.

Second, a different approach, avoiding rather than solving the problem:

Cooks44, you mentioned Samba, and that it might not like you. This makes me guess that the destination (JBOD on NAS) lives on a Windows machine. Is this correct? If yes, you could try rsync-ing to that directly, and avoid using the Samba mount. I routinely do that here at home. I initiate the rsync from the Linux side (by cron), so the Windows machines must be running sshd. Rync also works the other way.

/Quigi

chrism01 · 11-21-2007, 07:20 PM

y, direct rsync is simpler. Could simply be a permissions issue? ie can recv rsync delete the files and/or file perms are set r--r--r-- or equiv....

cooks44 · 11-21-2007, 11:57 PM

NAS is a Thecus NS5200, my understanding is it runs a "proprietory" version of linux on it. Unfortunately it doesn't run the rsync protocol, and being designed for windows use it only provides for samba connections (or so I recall - haven't checked what else is available in a while).

Haven't had a chance to revisit it as yet - our ISDN PRI has decided it wants to join the fun and is giving me an immense amount of grief. That takes priority so I'll be back to this issue soon.

Thanks all for keeping the possibilities up!

Cooks44.