LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-03-2008, 04:04 PM   #1
crazy8
Member
 
Registered: Nov 2006
Posts: 132

Rep: Reputation: 15
rsync questions


Ok I am a total n00b to rsync and I have finally incorporated some of it into a back up script we use where I work. But I do have a few questions about it all so here I go.
This is my script right now...
PHP Code:
mount /dev/sdb1 /mnt/backup
rsync 
--delete -avuH /etc /mnt/backup
tar 
-cvf /mnt/backup/etc.tar /mnt/backup/etc

rsync 
--delete -avuH //mnt/backup
rsync 
--delete -avuH /var/www /mnt/backup
rsync 
--delete -avuH /d2a /mnt/backup
rsync 
--delete -avuH /home /mnt/backup
rsync 
--delete -avuH /var/lib/mysql /mnt/backup
rsync 
--delete -avuH backup /mnt/backup
rsync 
--delete -avuH backup1 /mnt/backup

tar 
-cvf /mnt/backup/scooter.tar /mnt/backup//mnt/backup/var/www /mnt/backup/d2a mnt/backup/etc /mnt/backup/home /mnt/backup/var/lib/mysql /mnt/backup/backup /mnt/backup/backup1

rm 
-fr /mnt/backup//mnt/backup/var/www /mnt/backup/d2a /mnt/backup/home /mnt/backup/var/lib/mysql /mnt/backup/backup /mnt/backup/backup1

#cp /data/* /mnt/backup
#cp backup /mnt/backup
#cp backup1 /mnt/backup

mount win2:/ /win2 -o rsize=1024,wsize=1024
tar 
-cvf /mnt/backup/win2.tar /win2//win2/var/www /win2/etc

df 
-H
ls 
-al /mnt/backup
umount 
/win2
df 
-H
umount 
/mnt/backup
echo
echo 
Done with Backup
echo 
Now what this is supposed to do right now (the first run) is copy the specified directories from the server to the external drive then create the appropriate .tar files from them. After the .tar files are created then delete the folders off the external drive.
Now for the questions...
1) Im sure my rsync code could be cleaned up better somehow, any thoughts?

2) I also will need to update this external drive time to time so will "--delete" be enough to remove files from the drive that are no longer on the server, if not what else should be added?

3) Again, since this external drive will be updated is there a way I can still search the server for "brand new files" and "updated" files in specified directories and add them to my .tar files without running through the whole script? If so how.

4) And lastly (for now anyway) is anyone familiar enough with what I have here to "streamline" it and still be able to achieve what I am doing here?

Thank you all so very much for your help.
 
Old 04-03-2008, 07:49 PM   #2
blacky_5251
Member
 
Registered: Oct 2004
Location: Adelaide Hills, South Australia
Distribution: RHEL 5&6 CentOS 5, 6 & 7
Posts: 572

Rep: Reputation: 56
One of the benefits of rsync is its ability to just transfer file differences - this is especially helpful when transferring data between servers or networks. Because you're deleting all traces of the data from your external device after you've copied it, then there is nothing at the destination end for rsync to compare the source to - ergo everything is transferred, not just the differences.

Further, because you're deleting from /mnt/backup after you've copied it (with the rm -fr command), the --delete on your rsync commands is superfluous - again because there is nothing in the destination to be deleted if it no longer exists in the source. You can't delete what you haven't got

Unless you're prepared to retain your /mnt/backup/u directories etc., you won't obtain the speed benefit from just transferring file differences. If you're using rsync for other reasons, then drop the --delete from your command line as it is not doing anything.
 
Old 04-03-2008, 09:30 PM   #3
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,191

Rep: Reputation: 105Reputation: 105
If you really want to be using rsync and get the advantages of rsync, then don't remove those directories when you are done. rsync, on subsequent runs, with the --delete, will synchronize the server and the drive. Since this is your "backup", skip the tar.

On the other hand, if you want a collection of historical tar files, then just run the tar from the originals with the tar file going onto the external drive. No point in copying the files there and then taring them and removing them.
 
Old 04-03-2008, 10:01 PM   #4
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 7,183

Rep: Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212
You might find it advantageous to write this script in a readily-available programming language ... Perl, perhaps. Or PHP, or Ruby... Python anyone? In Linux-land, all of these are "at your fingertips" and why not take advantage of them?

You might also wish to look-around for existing Linux/Unix tools ... also free, no doubt ... which might do what you're setting out to do and already do it to a much more-refined stage of completion. If what you're ultimately looking for is "synchronization" or "replication," which basically you are, well... never forget this mantra:
Quote:
Dictum Ne Agas: Do Not Do A Thing Already Done.
What this actually means, in practice, is that you should aggressively look for examples of where "what you're actually trying to accomplish" has already been done. Whether the approach is identical or even comparable, or not. This means doing "research first," and "thinking outside the box" as you do so. Quite often that means busting-out of preconceived notions, and for a typical programmer-type that can be very hard to do.

"Programmers" (and hey, I'm one too...) tend to view everything they encounter as not only "a problem" but "a new problem." As though somehow they were the first humans on the planet to have encountered it. Undoubtedly-not-true, but "look at 'em, there they go, writing scripts before they even stop to think."

We can't do too much for the brain-dead Windows programmers who spend so much time fauning over such a ridiculously-limited programming environment, but this is Linux/Unix! A cornucopia of riches: where do we start? We start by looking for what we probably already have and don't even know about.

It does take practice.
 
Old 04-03-2008, 11:47 PM   #5
crazy8
Member
 
Registered: Nov 2006
Posts: 132

Original Poster
Rep: Reputation: 15
Thank you all for your help. All of this is very informative.
blacky_5251: Ill make the changes you are suggesting. Thanks for pointing that out. I was under the impression that --delete just deleted files/folders from the destination only if the particular file no longer exists on the source.

Quote:
choogendyk If you really want to be using rsync and get the advantages of rsync, then don't remove those directories when you are done
You know after I left work I did think of this.

Quote:
sundialsvcs You might find it advantageous to write this script in a readily-available programming language ... Perl, perhaps. Or PHP, or Ruby... Python anyone?
Well me being somewhat of a website developer/programmer I am lucky enough to know some PHP. However I dont know Perl, Ruby, or Python. So would there be any tutorials that you know of on how to do what I need to do in PHP or anything else?

Thank you all again for your help. I greatly appreciate it.
 
Old 04-04-2008, 09:26 AM   #6
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,191

Rep: Reputation: 105Reputation: 105
Interesting that in your original post the code segment was headed with "PHP Code:", but the code looks like it's all just basic shell commands.

I've programmed in probably 20 programming languages (and I'm not counting scripting environments), but I would opt for shell script in this case. I don't really see any requirements that don't fit. You can do it all from the command line, so just put it in a file and call it a script. I've done far more complicated backup "programs" as shell scripts.

There are collections of "recipes" for doing particular things with rsync online that might be worth looking at. You can use rsync with snapshots to efficiently store what was on your server at different points in time without having to do separate full backups each time. So, yeah, search the web. For example: http://samba.anu.edu.au/rsync/examples.html or http://www.mikerubel.org/computers/rsync_snapshots/
 
Old 04-04-2008, 02:27 PM   #7
crazy8
Member
 
Registered: Nov 2006
Posts: 132

Original Poster
Rep: Reputation: 15
Well here is the updated version of my script so far...looking any better? I am keeping the .tar stuff just because thats what my boss wants. What boss wants, boss gets.

################################################################################
# --delete-during = file deletions on the receiving side will be done as the
# transfer happens.
#
# -a = archive mode
# -v = verbose
# -u = update. This will skip any files on the destination that have a modified
# time that is newer than the source file.
#
# -t = time Will transfer modification times along with files and update them on# the remote system.
#
# -H = hard links This will look for hard linked files in the transfer and link
# together the corresponding files on the receiving side.
################################################################################

mount /dev/sdb1 /mnt/backup

rsync --delete-during -avutH /etc /mnt/backup
tar -cvf /mnt/backup/etc.tar /mnt/backup/etc

rsync --delete-during -avutH /u /mnt/backup
rsync --delete-during -avutH /var/www /mnt/backup
rsync --delete-during -avutH /d2a /mnt/backup
rsync --delete-during -avutH /home /mnt/backup
rsync --delete-during -avutH /var/lib/mysql /mnt/backup
rsync --delete-during -avutH backup /mnt/backup
rsync --delete-during -avutH backup1 /mnt/backup

tar -cvf /mnt/backup/scooter.tar /mnt/backup/u /mnt/backup/var/www /mnt/backup/d2a mnt/backup/etc /mnt/backup/home /mnt/backup/var/lib/mysql /mnt/backup/backup /mnt/backup/backup1


cp /data/* /mnt/backup
# cp backup /mnt/backup
# cp backup1 /mnt/backup

mount win2:/ /win2 -o rsize=1024,wsize=1024
tar -cvf /mnt/backup/win2.tar /win2/u /win2/var/www /win2/etc

date --utc >> backup_log
echo : Backup Complete >> backup_log

df -H
ls -al /mnt/backup
umount /win2
df -H
umount /mnt/backup
echo
echo Done with backup.

Thanks for all the help you guys.

I just thought of this now. As I have stated before my command/programming skills are somewhat limited but Im wondering something. When i run my script it has to go through 10's of thousands of files and runs for a very long time because of this. Is there a way to incorporate "touch" or some other command that could search all the directories much quicker and do the comparison of the dates between source and destination files?
Also once (if) new files are found and added to the external drive, is there a way to just "insert" ONLY the new files into the .tar in their appropriate spot? That way time could be saved by not having to recreate a new .tar every time the script runs.

Last edited by crazy8; 04-04-2008 at 02:57 PM.
 
Old 04-04-2008, 06:00 PM   #8
blacky_5251
Member
 
Registered: Oct 2004
Location: Adelaide Hills, South Australia
Distribution: RHEL 5&6 CentOS 5, 6 & 7
Posts: 572

Rep: Reputation: 56
rsync is very clever and can deal with large numbers of files easily - as long as you have a file at the other end to compare with. Now that you have removed the rm command, your /mnt/backup directory will (approximately) mirror the files you want to backup. The first time you run it will take some time (while it creates the "mirror"), but thereafter your execution times should tumble.

You can't insert into a tar archive.
 
Old 04-04-2008, 06:14 PM   #9
viron
Member
 
Registered: Dec 2005
Location: McKinney, Texas
Distribution: Slackware
Posts: 40

Rep: Reputation: 15
If the tar archives aren't being removed each time you run the script, you can use the following command to update your archives instead of re-taring the entire directories:

tar --update -v -f /mnt/backup/win2.tar /win2/u /win2/var/www /win2/etc

Of course this only works if you're updating an existing tar archive. It should be noted that this isn't ideal for creating backup's since it doesn't overwrite the existing file(s) and/or directory(s). It simply adds the newer file to the end of the archive.

Last edited by viron; 04-04-2008 at 06:17 PM.
 
Old 04-05-2008, 12:22 AM   #10
crazy8
Member
 
Registered: Nov 2006
Posts: 132

Original Poster
Rep: Reputation: 15
Well maybe I will just stick with what I have then. I do however have just one more question. Because of all the option in rsync I want to be sure I am also doing exactly what my boss is asking of me. The purpos of this back up drive (as im sure with most people) is that if the server totaly crashes we need backups of those directories as they contain the corporate site along with backups of all the computers on the network. So here is my question. With the options I am using with rsync right now, if our server TOTALY crashed tomorrow will the files on my external be complete and be what I need for a smooth recovery?

Thanks again
 
Old 04-05-2008, 12:42 AM   #11
blacky_5251
Member
 
Registered: Oct 2004
Location: Adelaide Hills, South Australia
Distribution: RHEL 5&6 CentOS 5, 6 & 7
Posts: 572

Rep: Reputation: 56
Bare metal or disaster recovery is a whole other topic. I suggest you search the site or internet for disaster recovery or bare metal recovery. Here are two LinuxQuestions.org threads I found:- http://www.linuxquestions.org/questi...ght=bare+metal and http://www.linuxquestions.org/questi...ght=bare+metal
 
Old 04-05-2008, 01:59 PM   #12
choogendyk
Senior Member
 
Registered: Aug 2007
Location: Massachusetts, USA
Distribution: Solaris 9 & 10, Mac OS X, Ubuntu Server
Posts: 1,191

Rep: Reputation: 105Reputation: 105
Quote:
Originally Posted by crazy8 View Post
I want to be sure I am also doing exactly what my boss is asking of me. The purpos of this back up drive (as im sure with most people) is that if the server totaly crashes we need backups of those directories as they contain the corporate site along with backups of all the computers on the network.
That kind of changes things.

For one thing, "doing exactly what my boss is asking" is generally a CYA kind of thing. It's often not the right thing.

So, on to the second part of the quote above. As blacky_5251 said, this is a whole 'nother topic. Backup & Recovery. Disaster Recovery. I don't know what your experience and training is, what your job is, or how many "all the computers on the network" is, or how that is getting onto the directories on the server, . . . . But, it sounds like you need to do some more research and reading in order to break out of the CYA position and take some responsibility for what really needs to be done.

Backup & Recovery is actually the title of a very good book by Curtis Preston, published by O'Reilly. If you can, get a copy of it. It also has a companion web site http://www.backupcentral.com. There are also some links here: http://www.linuxquestions.org/bookmarks/tags/backup.

blacky_5251 has given a foothold into the Disaster Recovery area. Typically, you'll want to integrate your backup and recovery implementation with your disaster recovery implementation. Some people take the approach for workstations and even some servers that if it dies, they'll rebuild it, install the base operating system and applications, and then recover the data. Some people combine disk imaging for disaster recovery with backups for data recovery. There's no one "shoe" or "style" that fits all.

Get your script right for now, but don't let it end there. Learn all you can and come up with a proposal that really suits your situation and that you think you have learned enough to implement.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Rsync Questions progrock Linux - Networking 2 11-14-2007 12:41 PM
Rsync server vs rsync over ssh humbletech99 Linux - Networking 1 10-18-2006 01:10 PM
rsync questions slack66 Linux - General 4 11-05-2005 01:31 AM
Windows Rsync Upload to Linux Rsync - permissions inspleak Linux - Software 0 10-12-2004 03:49 PM
rsync questions csross Linux - General 3 06-25-2004 11:10 AM


All times are GMT -5. The time now is 05:00 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration