[SOLVED] rsync "temp" files bigger than original

mfoley · 07-11-2022, 01:13 PM

Before I kill an rsync I've got running, I'll ask first ... I'm running the following rsync on host A:

Code:

rsync -rtv --partial /mnt/hd/QBK/* HOSTB:/mnt/2022Q2

It's been working; that is it has been copying files, but it appears to be "stuck" on one file. On the local (HOSTA) it is:

Code:

ls -lh /mnt/hd/QBK/
-rw-r--r--  1 root root  145G 2022-07-08 03:17 commonW10VM.zip

But on HOSTB I have:

Code:

ls -lah /mnt/2022Q2/
-rw-------  1 ohprs ohprs 105G 2022-07-11 14:08 .commonW10VM.zip.kcLR6M
-rw-------  1 ohprs ohprs 128G 2022-07-11 14:08 .commonW10VM.zip.oKzHFc
-rw-------  1 ohprs ohprs 126G 2022-07-11 14:08 .commonW10VM.zip.wjZQoH

As you can see, the temp destination files are a total of 359G whereas the source files is only 145G.

All three files on the target keep growing, so the rsync hasn't stopped, but it seems like it should have finished copying long ago. It's been running for 3+ days (although not on just this one file).

Is this normal? I'm considering killing the rsync and starting over with this file.

boughtonp · 07-11-2022, 04:31 PM

Quote:

Originally Posted by mfoley

As you can see, the temp destination files are a total of 359G whereas the source files is only 145G.

Might be a sparse file?

Rsync has a --sparse option for handling them.

Quote:

Originally Posted by mfoley

Before I kill an rsync I've got running

This is why I always run rsync with --partial option.

mfoley · 07-11-2022, 05:01 PM

Actually, it shouldn't be that sparse. It's a ZIP of a bz2 file, so it should be pretty compressed.

rknichols · 07-12-2022, 08:15 AM

It looks like you have 3 instances of the same rsync running concurrently and competing for CPU time and I/O bandwidth.

mfoley · 07-13-2022, 12:21 AM

Quote:

Originally Posted by rknichols

It looks like you have 3 instances of the same rsync running concurrently and competing for CPU time and I/O bandwidth.

Yes, it does look like that, but I don't really. I only ran one rsync command. It finally finished that file, then the next one started, and it too showed 3 destination work files. Not sure what's up with that. Maybe it's an rsync thing to copy parts of the file in parallel.

In any case, the all eventually finished. Maybe there was just that much stuff to copy!

rknichols · 07-13-2022, 03:58 PM

Quote:

Originally Posted by rknichols

It looks like you have 3 instances of the same rsync running concurrently and competing for CPU time and I/O bandwidth.

Quote:

Originally Posted by mfoley

Yes, it does look like that, but I don't really. I only ran one rsync command. It finally finished that file, then the next one started, and it too showed 3 destination work files. Not sure what's up with that. Maybe it's an rsync thing to copy parts of the file in parallel.

That's strange. I just tried an rsync with a ~140GB file and only saw a single temporary file being used. I tried it both with and without the "--partial" option to see if there was any difference. None. Now my destination "network" was "localhost:", so the transfer went fairly quickly (under 90 minutes). Perhaps you were having some network interruptions that forced rsync to resume an in-progress transfer.

boughtonp · 07-13-2022, 05:17 PM

A bit annoyed I didn't twig the three temp files before.

A single rsync with multiple growing temp files that total more the twice the target file size seems an unlikely bug - far more likely is three instances of rsync running.

If it occurs again see what "ps all | grep '[r]sync'" says.