Web Server Redundancy Help

illNino · 11-20-2008, 07:10 PM

Hey all,

Not sure exactly what section this goes in, but I am a Linux noob so thought I would just post it here. I am trying to come up with a decent redundancy plan for my web hosting business. Currently have 1 server at a data center which does our mail, dns, mysql, websites (apache), and if that where to go down we would be pretty screwed in terms of the websites.

We have obtained another server and trying to come up with a way to have very minimal downtime if something went wrong to our web server. We are also considering a VM solution, something like just rsyncing guests between the 2 servers. Would we need a central storage location to rsync from? Not exactly sure how it all works, but trying to figure it out.

Any advice would help, thanks.

rylan76 · 11-21-2008, 07:49 AM

Not sure about rsync and stuff, but I've done something similar with two servers with a very simple PHP script I wrote.

What basically happens is there is a cronjob (i. e. something that executes regularly every 24 hours) on the primary server that tars and bzips all important directories, calls mysqldump to dump the mysql databases, and then FTPs all this over to the secondary machine.

The idea being, if the primary goes down, I can have the center admins switch to the backup almost immediately (they'll just need to forward the traffic from the "old" IP to the new one) until a DNS update can propagate to relink the symbolic addresses with the "new" IP address. (Preferably I'd have them fix whatever went blooey on the primary system, then get it back as soon as possible to maintain redundancy).

Here's more or less what the bash script looks like that archives all the important stuff up and then dumps the vital tables from mysql:

Code:

mysqldump --set-variable=max_allowed_packet=4M --add-drop-table -uuser -ppass -hserver.com dbname > fwf_db_backup.sql

bzip2 /usr/home/fwfunn/fwf_db_backup.sql

mv /usr/home/fwfunn/fwf_db_backup.sql.bz2 public_html/backup_export/.

tar cvf fwf_files_backup_1.tar /usr/home/fwfunn/public_html/downloads/dynamic/*

bzip2 /usr/home/fwfunn/fwf_files_backup_1.tar

mv /usr/home/fwfunn/fwf_files_backup_1.tar.bz2 public_html/backup_export/.

tar cvf fwf_files_backup_2.tar /usr/home/fwfunn/public_html/images/dynamic/*

bzip2 /usr/home/fwfunn/fwf_files_backup_2.tar

mv /usr/home/fwfunn/fwf_files_backup_2.tar.bz2 public_html/backup_export/.

/home/httpd/cgi-bin/php5 /usr/home/fwfunn/public_html/send_backups.php

The last line above calls the php script that FTPs the files created above over to the secondary:

Code:

<?php
include 'cms/class.phpmailer.php';

$ftp_server = "backupserver.com";

if (($conn_id = ftp_connect($ftp_server)))
  {
    if (($login_result = ftp_login($conn_id,"username","pass")))
      {
	$file = "backup_export/fwf_db_backup.sql.bz2";
	$file_2 = "backup_export/fwf_files_backup_1.tar.bz2";
	$file_3 = "backup_export/fwf_files_backup_2.tar.bz2";

	if (ftp_put($conn_id,"public_html/auto_backups/" . basename($file),$file,FTP_BINARY))
	  {
	    if (ftp_put($conn_id,"public_html/auto_backups/" . basename($file_2),$file_2,FTP_BINARY))
	      {
		if (ftp_put($conn_id,"public_html/auto_backups/" . basename($file_3),$file_3,FTP_BINARY))
		  {
		    ftp_close($conn_id);

		    $mail = new PHPMailer();

		    $mail->From = "";

		    $mail->FromName = "FWF Backup Script";

		    $mail->AddAddress("user@somewhere.com");

		    $mail->Subject = "FWF Backup Script Success";

		    $msg = "fwf.co.za/backup_export/fwf_db_backup.sql.bz2 transferred succesfully to backup.com/auto_backups/fwf_db_backup.sql.bz2.\n\n";

		    $msg .= "fwf.co.za/backup_export/fwf_files_backup_1.sql.bz2 transferred succesfully to backup.com/auto_backups/fwf_files_backup_1.sql.bz2.\n\n";

		    $msg .= "fwf.co.za/backup_export/fwf_files_backup_2.sql.bz2 transferred succesfully to backup.com/auto_backups/fwf_files_backup_2.sql.bz2.\n\n";

		    $mail->Body = $msg;

		    if (!$mail->Send())
		      {
			echo "Backup script success email message cannot be sent.";
		      }
		  }

		else

		  {
		    $mail = new PHPMailer();

		    $mail->From = "";

		    $mail->FromName = "FWF Backup Script";

		    $mail->AddAddress("user@somewhere.com");

		    $mail->Subject = "FWF Backup Script Failure - File 2";

		    $mail->Body = "Failed during FWF file 2 transfer to backup.com, after successful connect and login!";

		    if (!$mail->Send())
		      {
			echo "Backup script FWF file transfer failure email message cannot be sent.";
		      }
		  }
	      }

	    else

	      {
		$mail = new PHPMailer();

		$mail->From = "";

		$mail->FromName = "FWF Backup Script";

		$mail->AddAddress("user@somewhere.com");

		$mail->Subject = "FWF Backup Script Failure - Files 1";

		$mail->Body = "Failed during FWF file 1 transfer to backup.com, after successful connect and login!";

		if (!$mail->Send())
		  {
		    echo "Backup script FWF file transfer failure email message cannot be sent.";
		  }
	      }
	  }

	else

	  {
	    $mail = new PHPMailer();

	    $mail->From = "";

	    $mail->FromName = "FWF Backup Script";

	    $mail->AddAddress("user@somewhere.com");

	    $mail->Subject = "FWF Backup Script Failure - DB";

	    $mail->Body = "Failed during DB transfer to backup.com, after successful connect and login!";

	    if (!$mail->Send())
	      {
		echo "Backup script DB transfer failure email message cannot be sent.";
	      }
	  }
      }

    else

      {
	$mail = new PHPMailer();

	$mail->From = "";

	$mail->FromName = "FWF Backup Script";

	$mail->AddAddress("user@somewhere.com");

	$mail->Subject = "FWF Backup Script Failure";

	$mail->Body = "Failed during login to backup.com, after successful connect!";

	if (!$mail->Send())
	  {
	    echo "Backup script login failure email message cannot be sent.";
	  }
      }
  }

 else

   {
     $mail = new PHPMailer();

     $mail->From = "";

     $mail->FromName = "FWF Backup Script";

     $mail->AddAddress("user@somewhere.com");

     $mail->Subject = "FWF Backup Script Failure";

     $mail->Body = "Failed during connect to backup.com!";

     if (!$mail->Send())
       {
	 echo "Backup script connect failure email message cannot be sent.";
       }
   }
?>

Hope this helps...

farslayer · 11-21-2008, 09:32 AM

Here are some other references you may find useful.

http://www.howtoforge.com/high_avail...apache_cluster

http://howtoforge.net/setting-up-a-l...-with-mysql5.1

http://dev.mysql.com/doc/refman/5.0/...ion-howto.html

http://www.howtoforge.com/mirroring_with_rsync

http://howtoforge.net/howtos/high-availability

illNino · 11-22-2008, 12:56 AM

Awesome, thanks for the info guys.

The priority really is having this automated in case something happens over night as we have clients across the board. Would it be possible to have something like an apache and mysql cluster going? Keeping in mind we are setting up VMware on both servers for a bit extra redundancy.

How long would it take the dns to propagate also if we did it with that php script?

rylan76 · 11-22-2008, 01:52 AM

Well DNS usually takes +- 24 hours in my experience. However, I have had some customers inside a large organisation where it sometimes took 48 hours. I suspect they have some form of "DNS cache" locally on their intranet, they always lagged behind the rest of the world in "seeing" a new server tied to a symbolic name.

Sure you can try the cluster etc. but the type of option I suggested (for us at least) is the cheapest, quickest, and easiest to implement...