LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 10-27-2010, 11:24 AM   #1
permalac
Member
 
Registered: Jul 2007
Location: Barcelona
Posts: 115

Rep: Reputation: 16
Multi-threaded rsync ?


Hello,

I want to copy a folder from one server to another, the folder is full of small files and it goes very slow because of that.

I've been looking around for a thing like a multi-treaded rsync to copy faster but I haven't found anything.

Can somebody help me on this?


thanks.
 
Old 10-27-2010, 11:30 AM   #2
fbobraga
Member
 
Registered: Jul 2010
Location: São Paulo - Brasil
Distribution: Debian 7 / Crunchbang 11
Posts: 229

Rep: Reputation: 41
it's a job to do only once? If so, there's my tip: zip/tar the files in the source, copy/move the package created, and extract it on the destination

Last edited by fbobraga; 10-27-2010 at 11:32 AM.
 
Old 10-27-2010, 11:50 AM   #3
permalac
Member
 
Registered: Jul 2007
Location: Barcelona
Posts: 115

Original Poster
Rep: Reputation: 16
its a 70gb directory in a disc without free space, so there is no space for the resulting tar file.
 
Old 10-27-2010, 12:15 PM   #4
fbobraga
Member
 
Registered: Jul 2010
Location: São Paulo - Brasil
Distribution: Debian 7 / Crunchbang 11
Posts: 229

Rep: Reputation: 41
Quote:
Originally Posted by permalac View Post
its a 70gb directory in a disc without free space, so there is no space for the resulting tar file.
You cannot use a external/USB HDD? - it would make the whole thing a lot easier
 
Old 10-27-2010, 04:44 PM   #5
TobiSGD
Moderator
 
Registered: Dec 2009
Location: Hanover, Germany
Distribution: Main: Gentoo Others: What fits the task
Posts: 15,592
Blog Entries: 2

Rep: Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047Reputation: 4047
You can use tar in conjunction with netcat to do this job. Have a look here.
 
Old 10-27-2010, 06:11 PM   #6
permalac
Member
 
Registered: Jul 2007
Location: Barcelona
Posts: 115

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by fbobraga View Post
You cannot use a external/USB HDD? - it would make the whole thing a lot easier

What happens if the server is not side by side with the coffee machine?

Sorry, just kidding, probably in 10 hours or so it will be done with rsync, I was just wondering if there is a way to parallelize rsync to use all the cores from a host.
 
Old 10-27-2010, 06:17 PM   #7
permalac
Member
 
Registered: Jul 2007
Location: Barcelona
Posts: 115

Original Poster
Rep: Reputation: 16
Quote:
Originally Posted by TobiSGD View Post
You can use tar in conjunction with netcat to do this job. Have a look here.

In this case I will have to copy all again in case of an updated file. Nice approach anyway.
 
  


Reply

Tags
linux, rsync


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Multi threaded Wine txm123 Linux - Newbie 2 08-14-2010 01:56 PM
related to multi threaded programming sky rulz Linux - General 5 05-12-2009 12:34 AM
apache: running multi-threaded or multi fork? Swakoo Linux - General 1 03-20-2008 07:18 AM
how to do waitpid() using a multi-threaded server? Thinking Linux - Software 1 04-13-2005 09:50 PM
Multi-Threaded C pragti Programming 1 06-01-2004 10:50 AM


All times are GMT -5. The time now is 06:13 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration