LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 07-08-2012, 08:23 AM   #1
Eclectic
LQ Newbie
 
Registered: Aug 2005
Location: Norwich, UK
Distribution: Gentoo
Posts: 10

Rep: Reputation: 0
sync solution


Hi, I have a remote web server (running Ubuntu) and a local server (running Gentoo). Im looking for a way to keep the files on the remote box synced with the local box.

There's a large database to keep synced, some images and some larger files that Id like backed up at a pre defined time. Ive tried rsync but its painfully slow, especially with the images.

So Ideally a fast alternative to rsync, perhaps SFTP, but is it possible to have multi threads running on SFTP when its running from a script? Then how does it keep check of the new files and ignore ones already pulled down?

Many thanks!
Nige
 
Old 07-08-2012, 08:31 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,692

Rep: Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274
I think you can do it in two steps: 1: rsync everything without images and 2: rsync only images. In the second step you can try "whole-copy", so rsync will not try to find matching blocks.
 
Old 07-08-2012, 10:43 AM   #3
Eclectic
LQ Newbie
 
Registered: Aug 2005
Location: Norwich, UK
Distribution: Gentoo
Posts: 10

Original Poster
Rep: Reputation: 0
Hmm, but rsync is soo slow. Remote has a ridiculous connection and I have 60mb here. Remote is easily capable of maxing mine so ideally in the middle of the night Id like it doing just that. I do it currently with an FTP client. Id like it to be set and forget.
 
Old 07-08-2012, 11:01 AM   #4
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,692

Rep: Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274
rsync is definitely planned for low bandwidth connections and to minimize the amount of the transferred data. Using -W (whole-file) will switch off rsync's optimizing algorithm and will transfer every file one by one (if required). You can give a try, probably it runs faster, it is quite similar to a simple ftp (just it will skip identical files)
 
1 members found this post helpful.
Old 07-08-2012, 11:08 AM   #5
Eclectic
LQ Newbie
 
Registered: Aug 2005
Location: Norwich, UK
Distribution: Gentoo
Posts: 10

Original Poster
Rep: Reputation: 0
Mnay thanks. Ill give it a try!
 
Old 07-08-2012, 11:26 AM   #6
Eclectic
LQ Newbie
 
Registered: Aug 2005
Location: Norwich, UK
Distribution: Gentoo
Posts: 10

Original Poster
Rep: Reputation: 0
So, its faster.

Quote:
6515442 100% 2.51MB/s 0:00:02 (xfer#10, to-check=820/845)
I can live with that, but would like something a touch faster.

Thanks again!!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Ubuntu Linux solution stack implementation, Part 4: Solution stack setup and integration LXer Syndicated Linux News 0 08-18-2010 03:50 AM
Why use an enterprise software solution vesus a diy (home made) solution checkmate3001 Linux - Server 1 12-16-2007 02:24 PM
Solution: Gentoo - emerge --sync fails with "syntax error" daniel.santos Linux - Software 3 03-13-2007 12:56 AM
My Palm Sync solution in Ubuntu djzanni Linux - Hardware 0 09-05-2006 06:45 PM
Sync for Samsung Sync master Naphoon Linux - Hardware 1 10-16-2004 06:20 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:09 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration