Need Help For SImple Backup
 

I have a 2 servers running RHEL5 and SOLARIS.

on RHEL5 i have a directory files (as my Source)
/tenant/tent1/trans/t*.txt
/tenant/tent2/trans/t*.txt
/tenant/tent3/trans/t*.txt
:
/tenant/tentN/trans/t*.txt


on remote SOLARIS i have the same directories(as my target Destination)

The things to do
- Automatically create backup copies to another folder (e.g /backup/<same directory tree>
- Script only process directories that have files inside
- Create log files containing list of files to be copied.
- Backup the files from source to destination
- source files must be deleted automatically
- Report files that are not move into a log file
- The log file should have a file name derive on system time and date (e.g. 2008-01-09_05:30:10pm-sent.txt)

Any help is gladly appreciated.

Thanks,

Jon

what have you tried so far?

rsync is a very good program for backing up/mirroring a directory tree of files from one machine to another machine, and for keeping the two machines "in sync."

Yes, rsync is the best that I have seen works for stuff like this.
You can configure it to automatically login from one machine to another using authentication keys to identify. Search the web for the steps to do this.
You can set up a cron job and get your nightly backup's done this way.

I am making your work more easy.

http://www.scrounge.org/linux/rsync.html

Dont forget to thank me for that..

thanks a lot for the fast replies
 

Thanks chrism, uks and shadow.

I have tried rsync together with ssh-pass to automate login.

Below is my script (some codes are long method =) coz i dont have much experience scripting). I hope someone can help optimize.

# Begin Code
#! /usr/bin/perl
my $logdir = "/application/logfiles/filesync/";
my $maindir="/tenant/";
my $backupdir = "/data/tenant/";
my $fbdir = "bk2";
my $dirlistfile = "dirlist.txt";
my $dfsdir = "dfs";
my $userid = "poller";
my $password = "cyanide235";
# Sub Routine Sections
sub trim($)
{
my $string = shift;
$string =~ s/^\s+//;
$string =~ s/\s+$//;
return $string;
};
# Left trim function to remove leading whitespace
sub ltrim($)
{
my $string = shift;
$string =~ s/^\s+//;
return $string;
};
# Right trim function to remove trailing whitespace
sub rtrim($)
{
my $string = shift;
$string =~ s/\s+$//;
return $string;
};

sub logo()
{
print("\n----------------------------------");
print("\n|TX File Synchronize Ver.1.0 |");
print("\n|Created By: Jonzkie |");
print("\n|Created Date: 06/01/2009 |");
print("\n----------------------------------\n");
};
logo();
# Step 1 Get Directory Listing from Host
$cmd="sshpass \-p 5678uhbv ssh kermit\@193\.41\.134\.10 \"sh \/switch_home\/kerm\/gendir\.sh\" 1\>$logdir" . "dirsync.log 2\>$logdir" . "dirsync.err";
system($cmd);
wait;
$cmd="sshpass \-p 5678uhbv scp kermit\@193\.41\.134\.10\:\/switch_home\/fmrag\/perldbi\/dirlist\.txt \. " . " 1\>\>$logdir" . "dirsync.log 2\>\>$logdir" . "dirsync.err";
system($cmd);
wait;
# End of Step 1

# Step 2 do an ls | wc -l for each directory on the list extracting thos directory with files
open(fdirlist,$dirlistfile) || die ("Error: Cannot find Tenants File");
if ( -d "$maindir")
{
}
else
{
print "Directory $maindir does not exist.Halting\n";
exit;
}
my $dfscnt=0;
my $tcmd = "echo \"\# Filtered \`date\`\" \> tosync.txt";
system($tcmd);
wait;
while ($line = <fdirlist>)
{
if ( $line=~/^\;/ ) {next;}
@row=split(/\|/, $line);
my $tenant_no = trim($row[0]);
my $shop_dir = trim($row[1]);
my $cat = trim($row[2]);
my $rsyncdir="";
my $lsdir = $maindir . $shop_dir . "\/trans\/";
if ($cat eq "DFS")
{
if ($dfscnt = 1)
{
next;
}
else
{
$lsdir = $maindir . $dfsdir . "\/trans\/";
$rsyncdir= $dfsdir . "\/trans\/";
}
}

if ($cat eq "TC")
{
$lsdir = $lsdir . $fbdir . "\/";
$rsyncdir = $shop_dir . "\/trans\/" . $fbdir . "\/";
}
else
{
$lsdir = $maindir . $shop_dir . "\/trans\/";
$rsyncdir = $shop_dir . "\/trans\/";
}

my $lsfile = trim($lsdir) . "t*.txt";
$cmd = qx(ls $lsfile 2\>err\.log \| wc -l);
chop $cmd;
wait;
if (($cmd * 1) > 0)
{
print "\n\-\-\-\-\-\--\-\-\-\-\-\n";
print $lsdir . " : " . $cmd . " files found\.\n";
my $tcmd = "echo \"$lsdir\|$cmd files \|$tenant_no\|$cat\" \>\> tosync.txt";
system($tcmd);
wait;
# $cmdcp = "cp $lsdir" . "\/t*.txt " . $backupdir$rsyncdir
# system($cmdcp);
# wait;
}
}
#my $tcmd = "echo \"\- \/\*\" \>\> tosync.txt";
#system($tcmd);
#wait;
# End of Step 2

# Step 3 Rsync the files indicated on the tosync.txt
# for future rsync -> my $rsynccmd = "rsync \-avpWzhtog \-\-include\-from\=tosync\.txt \-\-progress \-\-verbose root\@193\.41\.134\.197\:\/tenant\/ ";
open(fdirfile,"tosync.txt") || die ("Error: Cannot find Tenants File");
while ($dline = <fdirfile>)
{
if ( $dline=~/^\;/ ) {next;}
if ( $dline=~/^\#/ ) {next;}
@drow=split(/\|/, $dline);
my $rs_dir = $drow[0];
my $syncdt = `date +%F_%T`;
chomp $syncdt;
my $rsynccmd = "sshpass -p $password rsync \-avpWzhtog \-\-remove\-sent\-files $rs_dir $userid\@193\.41\.134\.197\:$rs_dir 1\>$logdir" . "$syncdt\-sent.log 2\>$logdir" . "$syncdt\-sent\.err";
print $rsynccmd . "\n";
system($rsynccmd);
wait;
}

# End of Step 3
# End Code


But the problem is that im having difficulty on the proper usage of the --include-from=File parameter of rsync thats why i parsed the file manually passing those directory name as parameter for rsync.

and 2nd thing im a little confused on how to use the same directory path programatically in rsync.

e.g. rsync -avz --include-from=tosync.txt /tenant/ 193.41.134.197:?<same path as my source>

Content of tosync.txt
+ abr_111_01/trans/t*.*
+ cc_617_29/trans/bk2/t*.*
:
+ df_02_0923/trans/t*.txt
- /*


in this case. i only want to copy all t*.* from the directory /tenant/abr_111_01/trans/ to the remote directory 193.41.134.197:/tenant/abr_111_01/trans/ and the rest is ignore. so how do i acomplish this using the code above and rsync?


Thanks,

Jon