LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 08-09-2006, 01:21 PM   #1
matts76
LQ Newbie
 
Registered: Aug 2006
Posts: 2

Rep: Reputation: 0
File sync between 2 servers in ~ realtime


I am looking for a solution to do a 1-way sync from a primary server to a secondary server more-or-less in realtime. It can be limited to specific directories since it doesn't need to sync EVERYTHING on the server. We've been using Unison on a 5-minute CRON. The problem is that now our file-base has grown to a point where the Unison process is chewing up our CPU to unacceptable levels. We've explored the possibility of SAN solutions, but the cost is prohibitave. Does anyone have any ideas for how to accomplish this? Thanks!
 
Old 08-09-2006, 02:42 PM   #2
marozsas
Senior Member
 
Registered: Dec 2005
Location: Campinas/SP - Brazil
Distribution: SuSE, RHEL, Fedora, Ubuntu
Posts: 1,393
Blog Entries: 1

Rep: Reputation: 64
Did you try NFS ? It is really good if your both servers are in the same high speed LAN, I mean, not connected through a slow WAN.
 
Old 08-09-2006, 03:15 PM   #3
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790
Are you sure its a single cron kick off chewing up your CPU?

That is to say have you verified the sync that kicked off 5 minutes earlier has completed BEFORE the next one kicks off? If not your issue may simply be a problem with accessing the same resources. I've seen this often where users have multiple copies of the same simple script running and can't figure out from top what is eating up CPU. What is eating up CPU is the contention from multiple runs of the script all contending for exactly the same resources.

If this is the case simply putting a check in the script to verify another copy of the script isn't arleady running might help.

Failing that my suggestion would be NFS as the previous person indicated. The downside to this is the file only exists in one place in reality - if that place goes away (from a disk failure for example) both sides will lose it. With an rsync you have the original and the copy.
 
Old 08-09-2006, 04:50 PM   #4
matts76
LQ Newbie
 
Registered: Aug 2006
Posts: 2

Original Poster
Rep: Reputation: 0
Thanks. We need it for high availability more than anything else. You are correct in that it IS running multiple instances of itself and killing the CPU, but we don't want to run it any less often for data protection reasons. After doing some research we're leaning towards DRBD. Do you guys have any experience with this tool? Anything we should watch out for? Thanks!
 
Old 08-09-2006, 05:35 PM   #5
IsaacKuo
Senior Member
 
Registered: Apr 2004
Location: Baton Rouge, Louisiana, USA
Distribution: Debian 4.0 Etch
Posts: 1,349

Rep: Reputation: 49
What operating systems are the two servers running, and how are they connected? I found that nfs file transfers are about twice as fast as samba file transfers.

Have you tried using rsync? I use it via nfs to keep backups of the data on my server (this means using "rsync" as a glorified efficient version of "cp", rather than messing with the rsync daemon and stuff). I don't know if it's performance is better than Unison, but I'm rather impressed by its efficiency on my home systems.
 
Old 08-10-2006, 02:57 AM   #6
jayakrishnan
Member
 
Registered: Feb 2002
Location: India
Distribution: Slacky 12.1, XP
Posts: 991

Rep: Reputation: 30
http://linux.maruhn.com/sec/mirrordir.html
 
Old 08-12-2006, 10:29 AM   #7
MensaWater
Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 6,028
Blog Entries: 5

Rep: Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790Reputation: 790
Quote:
Originally Posted by matts76
Thanks. We need it for high availability more than anything else. You are correct in that it IS running multiple instances of itself and killing the CPU, but we don't want to run it any less often for data protection reasons. After doing some research we're leaning towards DRBD. Do you guys have any experience with this tool? Anything we should watch out for? Thanks!
I think you missed the point in my post. If a second instance kicks off before the first one completes you're not really getting the file(s) as often as you think. The first one has to complete to free up the resources for the second one to run. Running it so often buys you nothing and as you've seen in terms of performance costs you plenty.

Instead of cron to do the rsync I'd suggest:

1) Create a script that does an infinite while loop:

while true do
rsync ... (whatever your rsync command line is)
sleep 300
done

2) Create another script that verifies the above is running and put this other script in cron (every 15 minutes would do I should think).

The first script does the rsync until the end of time because "true" is always true. The cron job just makes sure the first script restarts if it should fail for any reason (like after a system reboot).

The "sleep 300" says to wait 5 minutes after the first rsync completes until the next one starts. You don't even have to wait 5 minutes. Just putting in "sleep 60" waiting a minute will let it start a minute later. Even a "sleep 1" would be better than nothing.

Also do NOT put an ampersand (&) at the end of the rsync line. The & tells it to run asynchronously. You WANT it to be synchronous so that the next run follows the completion AND the sleep.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
filesystem in realtime sync bfay Linux - Software 10 02-17-2012 03:37 PM
Making 2 servers stay in instant sync andy7t Linux - Networking 2 12-19-2005 03:34 AM
Linux realtime file replication bretticus Linux - Software 4 11-03-2005 10:28 AM
File Sync? justinmc Linux - Networking 4 10-01-2005 02:40 PM
Large File Sync Problem climbingmerlin Linux - Software 2 09-23-2005 11:14 AM


All times are GMT -5. The time now is 05:24 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration