LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 04-04-2007, 11:57 AM   #1
masali
LQ Newbie
 
Registered: Apr 2007
Posts: 1

Rep: Reputation: 0
Transfer 1000000 files (130GB) server to server via shell


Hi guys!

I am having a problem. I have had some server problems with one (located in USA) of my two servers, and need to transfer all files to the other server (located in Germany). I've tried using Rsync, but it would take days to transfer this data (only get approx 500KB/sec). I have tried using several rsync sessions to transfer different folders to increase speed using parallel transfers, and it speeds up things alot. The problem is that folders are so different in sizes, so I it is not very practical to start a new rsync for every folder or so.

To the question: Is there any shell program that I can use to parallel transfer all these files (maybe 10-15 at a time) to speed up this process?

Thanks in advance
Masali
 
Old 04-04-2007, 12:09 PM   #2
treed
Member
 
Registered: Sep 2003
Posts: 45

Rep: Reputation: 15
Copies

Maybe you can use wget to get the files if you have a ftp server set up.

http://www.cyberciti.biz/tips/linux-...ownloader.html


You can also try uucp. This an unix to unix copy. You can do a man on uucp for all the options.

Finally another option is to try scp:

http://www.cyberciti.biz/faq/linux-o...work-computer/
 
Old 04-04-2007, 01:08 PM   #3
almatic
Member
 
Registered: Mar 2007
Distribution: Debian
Posts: 547

Rep: Reputation: 67
I'd pack the files first, this way you can also easily make sure that all is transfered correctly (md5checksum).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Backup MySQL databases, web server files to a FTP server automatically LXer Syndicated Linux News 0 08-11-2006 09:54 PM
How can I write a shell script to transfer files and keep permissions? LancerNZ Linux - Networking 1 02-24-2005 03:49 AM
How can I write a shell script to transfer files and keep permissions? LancerNZ Linux - Wireless Networking 0 02-24-2005 12:19 AM
How do I dial in to a remote server and transfer files? yilez Linux - Networking 1 05-18-2004 10:40 AM
Connection, login and upload files to ftp server via shell script Paulo Góes Linux - Networking 2 02-21-2004 01:01 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:20 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration