LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
LinkBack Search this Thread
Old 01-12-2009, 04:06 AM   #1
Jonzkie
LQ Newbie
 
Registered: Aug 2008
Posts: 2

Rep: Reputation: 0
Need Help For SImple Backup


I have a 2 servers running RHEL5 and SOLARIS.

on RHEL5 i have a directory files (as my Source)
/tenant/tent1/trans/t*.txt
/tenant/tent2/trans/t*.txt
/tenant/tent3/trans/t*.txt
:
/tenant/tentN/trans/t*.txt


on remote SOLARIS i have the same directories(as my target Destination)

The things to do
- Automatically create backup copies to another folder (e.g /backup/<same directory tree>
- Script only process directories that have files inside
- Create log files containing list of files to be copied.
- Backup the files from source to destination
- source files must be deleted automatically
- Report files that are not move into a log file
- The log file should have a file name derive on system time and date (e.g. 2008-01-09_05:30:10pm-sent.txt)

Any help is gladly appreciated.

Thanks,

Jon
 
Old 01-12-2009, 04:48 AM   #2
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.5
Posts: 16,086

Rep: Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995Reputation: 1995
what have you tried so far?
 
Old 01-12-2009, 05:06 AM   #3
your_shadow03
Senior Member
 
Registered: Jun 2008
Location: Germany
Distribution: Slackware
Posts: 1,429
Blog Entries: 6

Rep: Reputation: 51
rsync is a very good program for backing up/mirroring a directory tree of files from one machine to another machine, and for keeping the two machines "in sync."
 
Old 01-12-2009, 05:11 AM   #4
uks
Member
 
Registered: Jul 2007
Posts: 68

Rep: Reputation: 16
Yes, rsync is the best that I have seen works for stuff like this.
You can configure it to automatically login from one machine to another using authentication keys to identify. Search the web for the steps to do this.
You can set up a cron job and get your nightly backup's done this way.
 
Old 01-12-2009, 05:29 AM   #5
your_shadow03
Senior Member
 
Registered: Jun 2008
Location: Germany
Distribution: Slackware
Posts: 1,429
Blog Entries: 6

Rep: Reputation: 51
I am making your work more easy.

http://www.scrounge.org/linux/rsync.html

Dont forget to thank me for that..
 
Old 01-12-2009, 06:51 PM   #6
Jonzkie
LQ Newbie
 
Registered: Aug 2008
Posts: 2

Original Poster
Rep: Reputation: 0
thanks a lot for the fast replies

Thanks chrism, uks and shadow.

I have tried rsync together with ssh-pass to automate login.

Below is my script (some codes are long method =) coz i dont have much experience scripting). I hope someone can help optimize.

# Begin Code
#! /usr/bin/perl
my $logdir = "/application/logfiles/filesync/";
my $maindir="/tenant/";
my $backupdir = "/data/tenant/";
my $fbdir = "bk2";
my $dirlistfile = "dirlist.txt";
my $dfsdir = "dfs";
my $userid = "poller";
my $password = "cyanide235";
# Sub Routine Sections
sub trim($)
{
my $string = shift;
$string =~ s/^\s+//;
$string =~ s/\s+$//;
return $string;
};
# Left trim function to remove leading whitespace
sub ltrim($)
{
my $string = shift;
$string =~ s/^\s+//;
return $string;
};
# Right trim function to remove trailing whitespace
sub rtrim($)
{
my $string = shift;
$string =~ s/\s+$//;
return $string;
};

sub logo()
{
print("\n----------------------------------");
print("\n|TX File Synchronize Ver.1.0 |");
print("\n|Created By: Jonzkie |");
print("\n|Created Date: 06/01/2009 |");
print("\n----------------------------------\n");
};
logo();
# Step 1 Get Directory Listing from Host
$cmd="sshpass \-p 5678uhbv ssh kermit\@193\.41\.134\.10 \"sh \/switch_home\/kerm\/gendir\.sh\" 1\>$logdir" . "dirsync.log 2\>$logdir" . "dirsync.err";
system($cmd);
wait;
$cmd="sshpass \-p 5678uhbv scp kermit\@193\.41\.134\.10\:\/switch_home\/fmrag\/perldbi\/dirlist\.txt \. " . " 1\>\>$logdir" . "dirsync.log 2\>\>$logdir" . "dirsync.err";
system($cmd);
wait;
# End of Step 1

# Step 2 do an ls | wc -l for each directory on the list extracting thos directory with files
open(fdirlist,$dirlistfile) || die ("Error: Cannot find Tenants File");
if ( -d "$maindir")
{
}
else
{
print "Directory $maindir does not exist.Halting\n";
exit;
}
my $dfscnt=0;
my $tcmd = "echo \"\# Filtered \`date\`\" \> tosync.txt";
system($tcmd);
wait;
while ($line = <fdirlist>)
{
if ( $line=~/^\;/ ) {next;}
@row=split(/\|/, $line);
my $tenant_no = trim($row[0]);
my $shop_dir = trim($row[1]);
my $cat = trim($row[2]);
my $rsyncdir="";
my $lsdir = $maindir . $shop_dir . "\/trans\/";
if ($cat eq "DFS")
{
if ($dfscnt = 1)
{
next;
}
else
{
$lsdir = $maindir . $dfsdir . "\/trans\/";
$rsyncdir= $dfsdir . "\/trans\/";
}
}

if ($cat eq "TC")
{
$lsdir = $lsdir . $fbdir . "\/";
$rsyncdir = $shop_dir . "\/trans\/" . $fbdir . "\/";
}
else
{
$lsdir = $maindir . $shop_dir . "\/trans\/";
$rsyncdir = $shop_dir . "\/trans\/";
}

my $lsfile = trim($lsdir) . "t*.txt";
$cmd = qx(ls $lsfile 2\>err\.log \| wc -l);
chop $cmd;
wait;
if (($cmd * 1) > 0)
{
print "\n\-\-\-\-\-\--\-\-\-\-\-\n";
print $lsdir . " : " . $cmd . " files found\.\n";
my $tcmd = "echo \"$lsdir\|$cmd files \|$tenant_no\|$cat\" \>\> tosync.txt";
system($tcmd);
wait;
# $cmdcp = "cp $lsdir" . "\/t*.txt " . $backupdir$rsyncdir
# system($cmdcp);
# wait;
}
}
#my $tcmd = "echo \"\- \/\*\" \>\> tosync.txt";
#system($tcmd);
#wait;
# End of Step 2

# Step 3 Rsync the files indicated on the tosync.txt
# for future rsync -> my $rsynccmd = "rsync \-avpWzhtog \-\-include\-from\=tosync\.txt \-\-progress \-\-verbose root\@193\.41\.134\.197\:\/tenant\/ ";
open(fdirfile,"tosync.txt") || die ("Error: Cannot find Tenants File");
while ($dline = <fdirfile>)
{
if ( $dline=~/^\;/ ) {next;}
if ( $dline=~/^\#/ ) {next;}
@drow=split(/\|/, $dline);
my $rs_dir = $drow[0];
my $syncdt = `date +%F_%T`;
chomp $syncdt;
my $rsynccmd = "sshpass -p $password rsync \-avpWzhtog \-\-remove\-sent\-files $rs_dir $userid\@193\.41\.134\.197\:$rs_dir 1\>$logdir" . "$syncdt\-sent.log 2\>$logdir" . "$syncdt\-sent\.err";
print $rsynccmd . "\n";
system($rsynccmd);
wait;
}

# End of Step 3
# End Code


But the problem is that im having difficulty on the proper usage of the --include-from=File parameter of rsync thats why i parsed the file manually passing those directory name as parameter for rsync.

and 2nd thing im a little confused on how to use the same directory path programatically in rsync.

e.g. rsync -avz --include-from=tosync.txt /tenant/ 193.41.134.197:?<same path as my source>

Content of tosync.txt
+ abr_111_01/trans/t*.*
+ cc_617_29/trans/bk2/t*.*
:
+ df_02_0923/trans/t*.txt
- /*


in this case. i only want to copy all t*.* from the directory /tenant/abr_111_01/trans/ to the remote directory 193.41.134.197:/tenant/abr_111_01/trans/ and the rest is ignore. so how do i acomplish this using the code above and rsync?


Thanks,

Jon
 
  


Reply

Tags
file, rsync, sync, synchronization


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Simple backup script dotancohen Linux - Newbie 7 06-18-2006 11:53 PM
Simple backup system Azhrarn Linux - Software 1 09-16-2005 02:24 PM
Simple help with a backup script GraemeK Linux - Software 1 05-20-2005 11:43 PM
simple-backup-script ahmed4linux Linux - General 1 05-04-2005 07:10 PM
Simple way to backup my laptop enine Linux - General 2 02-04-2004 01:00 PM


All times are GMT -5. The time now is 01:54 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration