LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
LinkBack Search this Thread
Old 10-26-2011, 09:03 PM   #1
kukukachu
LQ Newbie
 
Registered: Jan 2010
Location: Jakarta, Indonesia
Distribution: ubuntu 11.10 64bit
Posts: 11

Rep: Reputation: 2
mirroring file system between 3 servers


i have a small network with three servers. the main server is located in Jakarta, and two "sub" servers are located in Bali and Batam.

For a few years now I have been using Unison to keep all the user files on these 3 systems in sync with one another. thus, if a user in Batam changes a file on their local server, then servers in Jakarta and Bali are also updated.

i just use consumer class 3Mbps internet connections for all this.

this system has worked sort of ok, but Unison has its limitations, bugs and problems.

is there any other way i can more reliably keep all these 3 servers in sync?

suggestions very much appreciated.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 10-27-2011, 03:13 AM   #2
deepak_message
Member
 
Registered: Oct 2007
Posts: 155

Rep: Reputation: 16
There is need to know, on which OS you are looking ?
becasue there are many softwares(freeware/Shareware) avalible for the sync. if you have linux you can use "rync" utility.

thanks,
 
Old 10-27-2011, 04:15 AM   #3
vasile002
Member
 
Registered: Oct 2011
Posts: 51

Rep: Reputation: Disabled
You could try using lsyncd:
http://code.google.com/p/lsyncd/
 
Old 10-27-2011, 05:33 AM   #4
kukukachu
LQ Newbie
 
Registered: Jan 2010
Location: Jakarta, Indonesia
Distribution: ubuntu 11.10 64bit
Posts: 11

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by deepak_message View Post
There is need to know, on which OS you are looking ?
becasue there are many softwares(freeware/Shareware) avalible for the sync. if you have linux you can use "rync" utility.
OS is Linux okusi0 2.6.35-28-server #50-Ubuntu SMP Fri Mar 18 18:59:25 UTC 2011 x86_64 GNU/Linux

i'm afraid rsync of itself is not appropriate for this situation, where you have three active servers that have to update each other, including new files, deleted files, moved files, and modified files.

i need a system like Unison, but which is more stable, less buggy, and maintained.
 
Old 10-27-2011, 05:36 AM   #5
kukukachu
LQ Newbie
 
Registered: Jan 2010
Location: Jakarta, Indonesia
Distribution: ubuntu 11.10 64bit
Posts: 11

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by vasile002 View Post
You could try using lsyncd:
http://code.google.com/p/lsyncd/
thanks vasile. yes, i was looking at that very program this morning. but the documentation, such as it is, is not very clear to me. it's also not clear how robust this system would be on often-flakey internet connections here in Indonesia.

does anyone have actual experience using lsyncd to synchronise files on multiple servers as outlined above??
 
Old 10-29-2011, 05:23 AM   #6
vasile002
Member
 
Registered: Oct 2011
Posts: 51

Rep: Reputation: Disabled
I am using it without problems. It uses rsync to sync the files. You need to set up passwordless rsa key auth between the servers you want synced and you can just run it as one command which makes it run as damoen. See lsyncd --help for all the options
 
Old 10-31-2011, 01:27 AM   #7
kukukachu
LQ Newbie
 
Registered: Jan 2010
Location: Jakarta, Indonesia
Distribution: ubuntu 11.10 64bit
Posts: 11

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by vasile002 View Post
I am using it without problems. It uses rsync to sync the files. You need to set up passwordless rsa key auth between the servers you want synced and you can just run it as one command which makes it run as damoen. See lsyncd --help for all the options
thanks vasile. i'm familiar with setting up passwordless rsa keys. i have to do this for Unison and other ssh programs. these are already set up on all my 3 servers.

one of the things that's not clear to me about lsyncd is this: Is it the case that you have to run lsyncd on all servers simultaneously? and is there any sort of special setup required for more than 2 servers?

Last edited by kukukachu; 10-31-2011 at 01:58 AM.
 
Old 10-31-2011, 03:31 AM   #8
vasile002
Member
 
Registered: Oct 2011
Posts: 51

Rep: Reputation: Disabled
Yes, it needs to run on all servers and there's nothing special required. Just set it up to sync the locations you need between all 3 servers. For 3 servers I believe you will need to run 2 lsyncd commands on each
 
Old 10-31-2011, 05:12 AM   #9
kukukachu
LQ Newbie
 
Registered: Jan 2010
Location: Jakarta, Indonesia
Distribution: ubuntu 11.10 64bit
Posts: 11

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by vasile002 View Post
For 3 servers I believe you will need to run 2 lsyncd commands on each
that sounds like a recipe for a "deadly embrace"! anyway, i guess there's nothing for it but to give it a go and see what happens!
 
Old 11-01-2011, 05:05 AM   #10
vasile002
Member
 
Registered: Oct 2011
Posts: 51

Rep: Reputation: Disabled
Quote:
Originally Posted by kukukachu View Post
that sounds like a recipe for a "deadly embrace"! anyway, i guess there's nothing for it but to give it a go and see what happens!
right, anyway nothing bad can happen
 
Old 11-06-2011, 02:27 AM   #11
kukukachu
LQ Newbie
 
Registered: Jan 2010
Location: Jakarta, Indonesia
Distribution: ubuntu 11.10 64bit
Posts: 11

Original Poster
Rep: Reputation: 2
Quote:
Originally Posted by vasile002 View Post
right, anyway nothing bad can happen
hmmmm .... ok.

here's what i've done. keep in mind that i'm still testing, and i'm completely unsure of its robustness.

i have 3 servers. one in Jakarta, one in Bali, and one in Batam. they are all connected to the internet on flaky, consumer-grade 3Mbps connections.

i'm assuming use of an Ubuntu server here; you rpm heads should be smart enough to make the relevant adjustments. also, if you want to use lsyncd -ryncssh, then you will need version 2 or higher, not the 1.x provided on the ubuntu repository. this means getting source code and compiling it yourself. don't panic. this is fairly painless if you have installed all the required dependencies, and this is done within the installation script below.

as root, install the following on each of the servers. (i put this in a file called installlsync.sh.)
Code:
#!/bin/bash

# make sure you have all the files required to compile from source
apt-get -y install build-essential gcc lua5.1 liblua5.1-dev liblua5.1-0 checkinstall

# make sure you have inotify installed.  
apt-get -y install inotify-tools

# this is the standard place for compiling programs from source.
mkdir /usr/local/src/lsyncd
cd /usr/local/src/lsyncd


# download newest lsyncd from lsyncd.googlecode.com (lsyncd-2.0.5.tar.gz)
# you may wish to check for newer versions.  i'm assuming 2.0.5 here.
wget http://lsyncd.googlecode.com/files/lsyncd-2.0.5.tar.gz

# unpack the code
tar xzvf lsyncd-2.0.5.tar.gz

cd lsyncd-2.0.5
./configure
make
make install
checkinstall -y

# note: if you are not using a 64bit server, then deb file 
# below may have a different name
dpkg -i lsyncd_2.0.5-1_amd64.deb

# remember copy the executable to somewhere useful!
cp lsyncd /usr/bin
once successfully installed, you then need to get lsyncd running on all three servers.

the FIRST thing you MUST do is to manually synchronise all the folders on each of the servers. if you don't do this first, you WILL lose files. trust me. i know these things.

so do something like this from your "master" serverJakarta:
Code:
cd /myfolder
rsync -avz . root@serverBali:/myfolder
rsync -avz . root@serverBatam:/myfolder
once everything is 100% in sync, i install this little script on each of the servers to automate/standardise the lsync setup on each machine: (call this script lsyncdgen.sh)
Code:
#!/bin/bash

# get rid of any instances of lsyncd currently running 
# (this assumes there is no activity on your system when you run this!)
killall lsyncd

# nominate a working directory where you are going to put this script 
# and the config file this script generates
wdir=/root/syncfile

# get the hostname of the server this script is running on
thishost=$(cat /etc/hostname)

# now we want to figure out which server we wish to synchronise...
# we have to go around in a circle to avoid deadly embraces ...
if [[ "$thishost" == "serverJakarta" ]]; then
    dst=serverBali
elif [[ "$thishost" == "serverBali" ]]; then
    dst=serverBatam
elif [[ "$thishost" == "serverBatam" ]]; then
    dst=serverJakarta
else
	# you messed up!
	echo "$thishost not on my list!"
	exit 1
fi


# let's make the lua config file for this server ...
echo "-- lsyncd config file for this server ($thishost)" > $wdir/myconfig.lua
# i'm setting up the log files for somewhere where i can find them
echo 'settings = { 
    logfile="/tmp/lsyncd.log", statusFile="/tmp/lsyncd.status", 
}' >> $wdir/myconfig.lua

# these are the rsync options you probably need:
opts="-aluz"

dir=/myfolder/
echo "sync{default.rsyncssh, source=\"$dir\", host=\"$dst\", targetdir=\"$dir\", rsyncOpts=\"$opts\", excludeFrom=\"$wdir/exclude\"}" >> $wdir
echo "" >> $wdir/myconfig.lua

#for each additional folder you want synchronised, do this code: 
#---
dir=/myfolder2/
echo "sync{default.rsyncssh, source=\"$dir\", host=\"$dst\", targetdir=\"$dir\", rsyncOpts=\"$opts\", excludeFrom=\"$wdir/exclude\"}" >> $wdir
echo "" >> $wdir/myconfig.lua
#---


# run lsyncd!
lsyncd  $wdir/myconfig.lua &
now you almost certainly want to exclude certain files from being synchronised. make a file called "exclude" and place in the same directory as the above script:
Code:
*~
~*
.*
that's it. keep an eye on the log files to see what is going on. i've only been using this a few hours, so have not fully tested it. i would be grateful for any input from anyone who has more experience using lsyncd!

NB: remember to ensure that your system can autologin to the destination server using ssh. use this script if this stuff confuses you. press Enter when/if you are presented with a passphrase prompt:
Code:
#!/bin/bash
# sshautologin.sh
if [ -z $1 ]; then
	echo "require destination username and servername"
	exit 1
fi
if [ -z $2 ]; then
	echo "require destination servername"
	exit 1
fi
cd ~
# if you already have a key then don't generate it again
if [ ! -f ~/.ssh/id_dsa.pub ]; then
	# no key! generate one!
	ssh-keygen -t dsa
	chmod 700 ~/.ssh
fi
cat ~/.ssh/id_dsa.pub | ssh -l $1 $2 'cat >> ~/.ssh/authorized_keys'

Last edited by kukukachu; 11-06-2011 at 03:52 AM.
 
2 members found this post helpful.
Old 11-09-2011, 10:43 AM   #12
vasile002
Member
 
Registered: Oct 2011
Posts: 51

Rep: Reputation: Disabled
nice one, thanks for posting detailed steps of how you did it
 
Old 09-01-2013, 12:29 PM   #13
wspivak
LQ Newbie
 
Registered: Mar 2008
Posts: 4

Rep: Reputation: 0
Lsyncd e-mail & mysql

I have a server (primary) that needs to duplicated for business continuity issues. If I use Lsyncd, can I duplicate users, logins, e-mail and Mysql databases on the Target (secondary) Server, this way if the primary looses the internet, I can change the dns and be up and running? Obviously the e-mail files need to be sync'd with the correct permissions.

IF this is not the product to accomplish this, anyone have any other ideas?

Thanks for the assistance.

Wayne
 
  


Reply

Tags
mirroring, server, synchronization


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Simple mirroring networked file system? pileofrogs Linux - Server 3 04-14-2010 09:44 PM
Unioning and mirroring file system over the internet cantab Linux - Server 2 04-06-2010 05:07 PM
Mirroring Slackware 12.1 servers Zafoid Linux - Newbie 3 12-10-2008 05:48 AM
Mirroring Linux Servers JohnLinx Linux - General 9 01-22-2004 07:44 PM
mirroring servers rkirkley Linux - Networking 0 10-08-2001 12:21 PM


All times are GMT -5. The time now is 02:49 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration