Multi-threaded rsync ?

permalac · 10-27-2010, 11:24 AM

Hello,

I want to copy a folder from one server to another, the folder is full of small files and it goes very slow because of that.

I've been looking around for a thing like a multi-treaded rsync to copy faster but I haven't found anything.

Can somebody help me on this?

thanks.

fbobraga · 10-27-2010, 11:30 AM

it's a job to do only once? If so, there's my tip: zip/tar the files in the source, copy/move the package created, and extract it on the destination

permalac · 10-27-2010, 11:50 AM

its a 70gb directory in a disc without free space, so there is no space for the resulting tar file.

fbobraga · 10-27-2010, 12:15 PM

Quote:

Originally Posted by permalac

its a 70gb directory in a disc without free space, so there is no space for the resulting tar file.

You cannot use a external/USB HDD? - it would make the whole thing a lot easier

TobiSGD · 10-27-2010, 04:44 PM

You can use tar in conjunction with netcat to do this job. Have a look here.

permalac · 10-27-2010, 06:11 PM

Quote:

Originally Posted by fbobraga

You cannot use a external/USB HDD? - it would make the whole thing a lot easier

What happens if the server is not side by side with the coffee machine?

Sorry, just kidding, probably in 10 hours or so it will be done with rsync, I was just wondering if there is a way to parallelize rsync to use all the cores from a host.

permalac · 10-27-2010, 06:17 PM

Quote:

Originally Posted by TobiSGD

You can use tar in conjunction with netcat to do this job. Have a look here.

In this case I will have to copy all again in case of an updated file. Nice approach anyway.