In Mint, on a GB ethernet connection, I am copying a large amount of data (around 800 GB) from a server to a local computer using rysnc. One directory on the server contains some large tar files (each one about 1 GB in size) that archived an entire (now-defunct) computer. The problem is that when rsync copies these tar files, it slows down to a crawl. It goes so slow, in fact--like 15 minutes per file--that it's making the job take a huge amount of time.

Does anyone know why this is happening, and if there is some switch in rsync that could prevent it? I suspect that rsync is recursing through the tar files and doing some type of checksum or something. All I want, though, is the files copied. Those tar files aren't even that important, so if there is a way to simply copy them byte by byte, that would be great.

If the files do not currently exist at the destination, rsync should be fast. If the files do exist at the destination, using the "-c" option will slow the startup considerably since the checksums of the source and destination files need to be calculated and compared.

The other thing that can slow down the transfer is a weak CPU at the source compressing the files for transfer. Since compression is the default, consider using the "--no-z" (--no-compression) option to turn it off, especially if the tar files are mostly incompressible anyway, or if your network is faster than the CPU doing the compression. (Compression is by default disabled for files with certain known incompressible extensions, which include ".gz", ".tgz", and ".tbz".)

Thanks, rknichols. The CPU is rather weak, so I am trying it with compression turned off, though as you say, the tar files (which have a gz extensuion), are not compressed by rsync anyway.

Compression has never been the default in my experience, is this something new with Ubuntu/Mint?

You're right. Scratch that. I'm getting my protocols mixed up again.

It' s not compression, its encryption. Rsync runs over ssh. If you run top while copying you might see an encryption process or SSH using much CPU time. If you see a high percentage of %wa (third line from the top) you have a network or disk problem.

jlinkels

There should be some posts on this exact tar rsync deal on large files. Think someone said it needed to do the entire file at a shot so you end up consuming all your ram and into swap space. Might double check that to be sure.


Could exclude those larger files and just copy them. Should fix the speed.