LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 07-11-2007, 05:41 PM   #1
jcorrea920
LQ Newbie
 
Registered: Jun 2006
Location: Oregon, USA
Distribution: Ubuntu / Fedora
Posts: 14

Rep: Reputation: 0
Question Two-way mirroring of Web Servers Sites


I have a Load Balanced High-Availability Web Cluster with 3 nodes. Therefore all 3 nodes must have the identical websites otherwise there will be inconsitencies with what appears on the browser with each http request. To keep the web sites consitent on all web servers I originally used rsync in a circular loop which caused many problems when files were created, deleted, or modified on a particular web server node. As you can imagine some changes were overwritten since rsync is one-way mirroring software. To make matters worse in order to get rsync to run automatically I had set up cronjobs to run every 5 minutes. Can you imagine someone uploading a file to one of these nodes and having to wait up to 10 minutes for them to replicate to the other nodes, and that is if it wasn't overwritten/deleted by the rsync process itself.

So I started to experiment with Unison. It achieves the two-way mirroring I need, but not the continuos part. It has a --repeat option that can be used for continuos replication but it does not operate in the background so it locks up your terminal window, for servers that have no GUI installed and administered remotely would eventually have the problem that the connection will time out not to mention the security risk of leaving a remote terminal window open (CTRL-C breaks the command and grants access to the shell ).

As a small band-aid solution I have created cronjobs to run unison on one of the nodes as it syncs with the other two every minute (Hub and Spoke topology as recommended by Unison docs). But still changes may take as long as 2 minutes to propagate to all nodes. Recently Unison has cease to exist in development and there is no support for it.

What I need is continuos two-way mirroring for these three nodes so that files on the websites are consitently the same. Either a daemon or background process that can achieve this will be greatly helpful. I was looking into Pavuk but the commands are horribly long and when I sent a message on their mailing list about it having two-way mirroring capabilities I received no reply, after 3 weeks!

All machines (3 nodes + 2 load-balancers):

Ubuntu 6.06 Server (no X-Windows/GUI)
Apache 2
Mysql 5
Php 5

Any suggestions or help would be greatly appreciated. Thanks for your support.

Jorge
 
Old 07-11-2007, 06:47 PM   #2
lineman60
Member
 
Registered: Oct 2003
Location: New Mexico
Distribution: CentOS, Ubnutu,
Posts: 35

Rep: Reputation: 15
1) try backgrounding the prosses using an ampersand (&)

2) howto forge has a nice set up useing clusted web servers http://www.howtoforge.com/high_avail...che_cluster_p4
3) what about a NFS, on NFS shared to all 3 webservers with Rsync backup of the data?

Last edited by lineman60; 07-11-2007 at 06:58 PM.
 
Old 07-15-2007, 06:35 PM   #3
jcorrea920
LQ Newbie
 
Registered: Jun 2006
Location: Oregon, USA
Distribution: Ubuntu / Fedora
Posts: 14

Original Poster
Rep: Reputation: 0
Thumbs down How could NFS help?

Quote:
Originally Posted by lineman60
1) try backgrounding the prosses using an ampersand (&)

2) howto forge has a nice set up useing clusted web servers http://www.howtoforge.com/high_avail...che_cluster_p4
3) what about a NFS, on NFS shared to all 3 webservers with Rsync backup of the data?
lineman60,

1) Tried & at the end of the unison command with the -repeat 002 option, it fails miserably. Fills the terminal window with all kinds of output even though I have also used -silent options as well. Did not work for unison. Maybe there is another way to get unison to work constently.

2) how to forge has some great stuff but all they recommend for replication among a web cluster is rsync. Which as I said before only uses one way synchronization, so it is limited to a max of two nodes, and I have three.

3) NFS I don't know much about I will look into it, but isn't that used for network shares as far as a file server with one-way remote replication?

Thanks to all who have tried to help, looking for simple replication solution. I know there is one out there. Feedback is greatly appreciated.

Patiently working on the problem,

J
 
Old 07-15-2007, 10:22 PM   #4
slimm609
Member
 
Registered: May 2007
Location: Chas, SC
Distribution: slackware, gentoo, fedora, LFS, sidewinder G2, solaris, FreeBSD, RHEL, SUSE, Backtrack
Posts: 428

Rep: Reputation: 65
add an entry to rc.local

/path/to/file -repeat 002


and it will start when the system starts
 
Old 07-15-2007, 11:14 PM   #5
p_s_shah
Member
 
Registered: Mar 2005
Location: India
Distribution: RHEL 3/4, Solaris 8/9/10, Fedora 4/8, Redhat Linux 9
Posts: 228
Blog Entries: 1

Rep: Reputation: 34
Also redirect output of unison command to some file, so you can check for conflicts.
Code:
unison ... >> /tmp/logfile
Don't forget to delete the log file periodically.
 
Old 07-16-2007, 10:25 PM   #6
jcorrea920
LQ Newbie
 
Registered: Jun 2006
Location: Oregon, USA
Distribution: Ubuntu / Fedora
Posts: 14

Original Poster
Rep: Reputation: 0
Question Great Suggestions but...

p_s_shah wrote:
Quote:
Also redirect output of unison command to some file, so you can check for conflicts.

Code:
Code:
unison ... >> /tmp/logfile
Don't forget to delete the log file periodically.
I tried the redirection to a > /dev/null and the terminal still gets filled up with garbage on all the conflicts, which at this point are many. So >> to a log file will probably do the same although I will try it for kicks this week.

slimm609 wrote:
Quote:
add an entry to rc.local

/path/to/file -repeat 002


and it will start when the system starts
what exactly is
Quote:
/path/to/file
I think you are refering to the unison command but I am not really sure.
 
Old 07-16-2007, 10:37 PM   #7
p_s_shah
Member
 
Registered: Mar 2005
Location: India
Distribution: RHEL 3/4, Solaris 8/9/10, Fedora 4/8, Redhat Linux 9
Posts: 228
Blog Entries: 1

Rep: Reputation: 34
Code:
/path/to/unison <options> 2>&1 > /tmp/logfile 
associates file descriptor 2 (standard error) with the same file as file descriptor 1 (standard output), then redirects both of them to /tmp/logfile.

Update if you still find problems.
 
Old 07-29-2007, 04:46 PM   #8
jcorrea920
LQ Newbie
 
Registered: Jun 2006
Location: Oregon, USA
Distribution: Ubuntu / Fedora
Posts: 14

Original Poster
Rep: Reputation: 0
Thumbs up [SOLVED] Unison works...

Thanks guys you were all a great help. So as suggested
by p_s_shah:
Quote:
Code:
/path/to/unison <options> 2>&1 > /tmp/logfile associates file
descriptor 2 (standard error) with the same file as file descriptor 1 (standard output), then redirects both of them to /tmp/logfile.
With the combination of & background command it seems to work every 2 seconds. For newbies the final code is:

Code:
path/to/unison -options -repeat 002 2>&1 > /tmp/logfile &
Also, I solved all conflicts reported into the logfile before actually starting the repetition which can be troubleshooted by simply removing -repeat 002 and the final & from the command. Then when I was error free I went ahead and went to production with the command. Thanks everyone...
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Mirroring Sites Setup Question dchwebphils Linux - General 0 04-17-2004 11:54 AM
Mirroring Linux Servers JohnLinx Linux - General 9 01-22-2004 07:44 PM
Web mirroring program UltimaGuy Linux - Newbie 3 09-30-2003 08:42 AM
Mirroring servers using RSYNC RKris Linux - Networking 1 05-15-2003 06:26 AM
mirroring servers rkirkley Linux - Networking 0 10-08-2001 12:21 PM


All times are GMT -5. The time now is 12:32 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration