LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (http://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   Backup script to write to file? (http://www.linuxquestions.org/questions/linux-newbie-8/backup-script-to-write-to-file-401677/)

ctroyp 01-10-2006 12:36 PM

Backup script to write to file?
 
Hello, I have already created a backup script that dd's hda to hdb. The script is placed in /etc/cron.daily. Within the script, I write the current date and time to a backup log, run the dd command, then I write another date and time to the same log to show how long the backup took.

I have two questions:
1)When running the dd command manuall in the shell, it returns the results of how much information was read from hda and how much information was written to hdb. I would like to write these results to the same log file along with the date and times that are already being written. I am unsure how to do this... Could someone help?

2)I would also like to calculate the elapsed time using the two dates and write it to the log file as well. This would allow me to monitor the time it takes to run the backup as the data grows. Any idea on how to create the calculation? I looked this up on google, but couldn't find what I was looking for.

Thanks in advance for any help offered.

ask.jackal 01-10-2006 12:43 PM

1. Redirect standard error to a file like below.

dd if=/dev/null of=/tmp/test 2>logfile

2. Not sure of any easy method. You may have to use ur prgmming skills !!

ctroyp 01-10-2006 12:48 PM

Quote:

Originally Posted by ask.jackal
1. Redirect standard error to a file like below.

dd if=/dev/null of=/tmp/test 2>logfile

Excellent! You made that easy...

Quote:

Originally Posted by ask.jackal
2. Not sure of any easy method. You may have to use ur prgmming skills !!

That's the problem, being a newbie, I have been pulling my hair out. I'm not bald yet so I guess I'll keep trying.

ctroyp 01-10-2006 12:53 PM

When using the redirect, will it overwrite the file? I want to append the log file so should I use this?
Code:

dd if=/dev/null of=/tmp/test 2>>logfile

ctroyp 01-10-2006 01:00 PM

Quote:

Originally Posted by ctroyp
When using the redirect, will it overwrite the file? I want to append the log file so should I use this?
Code:

dd if=/dev/null of=/tmp/test 2>>logfile

Disregard, I found the answer--'yes'.

jrdioko 01-10-2006 02:55 PM

And if you want all the output that would normally be spit out in the terminal, not just the error stream, use ">>" instead of "2>>".

haertig 01-10-2006 03:07 PM

Quote:

Originally Posted by jrdioko
And if you want all the output that would normally be spit out in the terminal, not just the error stream, use ">>" instead of "2>>".

That would get you stdout, but not stderr. To get both:
Code:

dd if=/dev/null of=/tmp/test >>logfile 2>&1
To SEE it on your screen, and LOG it to a file at the same time:
Code:

dd if=/dev/null of=/tmp/test 2>&1 | tee -a logfile

haertig 01-10-2006 03:23 PM

Aside from your main question of how to log program output, using dd to backup a filesystem while that filesystem is mounted and actively being used will not get you a good clean backup copy. So when you're dd'ing an entire harddisk as in your example, you don't want to be running out of any OS installed on that disk. Things would be changing on disk WHILE you're dd'ing, resulting in inconsistancies at minimum, something totally unuseable worst case. To dd a disk, boot Knoppix or some other live CD. To dd a partition, first unmount the filesystem using that partition, or at least make sure it's a stable not-normally-changing filesystem if you still feel adventurous. For something as important as a backup, I wouldn't recommend going the adventurous route. You might be surprized later when you actually need that backup!

jrdioko 01-10-2006 03:26 PM

Quote:

Originally Posted by haertig
That would get you stdout, but not stderr.

Interesting, never did read up on my output streams in any depth. I stand corrected.

ctroyp 01-10-2006 03:53 PM

Quote:

Originally Posted by haertig
Aside from your main question of how to log program output, using dd to backup a filesystem while that filesystem is mounted and actively being used will not get you a good clean backup copy. So when you're dd'ing an entire harddisk as in your example, you don't want to be running out of any OS installed on that disk. Things would be changing on disk WHILE you're dd'ing, resulting in inconsistancies at minimum, something totally unuseable worst case. To dd a disk, boot Knoppix or some other live CD. To dd a partition, first unmount the filesystem using that partition, or at least make sure it's a stable not-normally-changing filesystem if you still feel adventurous. For something as important as a backup, I wouldn't recommend going the adventurous route. You might be surprized later when you actually need that backup!

I am going after a new server w/RAID, backup tapes, etc. I am doing this as a temporary means to backup (what I can get) my system. I am unaware of another way to backup files on a webserver that is constantly working. Plus, I cannot take the server down to do a dd one time. Files change every minute and I have to backup daily at minimum.

Thanks for the feedback and feel free to give me any other ideas.

Also, does anyone have an idea on the elapsed time calculation?

haertig 01-10-2006 04:02 PM

Quote:

Originally Posted by ctroyp
Thanks for the feedback and feel free to give me any other ideas.

Use "rsync" to do these type of live backups. You can get fancy with excludes and all kinds of other stuff, but basically it's just something like:
Code:

rsync -va /usr /other_disk/other_dir
If files go changing out from under rsync, it will "do the right thing" (TM) ;-)

ctroyp 01-10-2006 05:04 PM

Quote:

Originally Posted by haertig
Use "rsync" to do these type of live backups. You can get fancy with excludes and all kinds of other stuff, but basically it's just something like:
Code:

rsync -va /usr /other_disk/other_dir
If files go changing out from under rsync, it will "do the right thing" (TM) ;-)

Okay, thanks a lot. I'll check into it. One question though...will the drive rsync'd to be a bootable drive?

haertig 01-10-2006 06:07 PM

rsync copies files. It does not create disk images. So no, your backup would not be bootable. But I think you would be in better shape to restore from rsync backups rather than with only the live dd's you have now. Do both - your live dd just may work, by shear dumb luck. To do a bare metal recovery you would typically need to first create a minimally bootable installation - either from your dd disk image (if it worked!) or by installing the minimal OS from scratch. But then you could bring this recovery system back up to full capabilities from your rsync backups.

Don't consider RAID a backup. It's a fallback, not a backup. You can recover an accidently deleted file from backups, but not from RAID. If a disk goes bad, a set of backups will not keep your server up and running during the fault, but RAID might. Two different things. I'm sure you know this already. But you'd be surprized how many people think RAID is a substitute for backups. If your webserver really must stay up 24x7 you should investigate using multiple computers with some type of IP takeover scheme (DNS changes can do some of this, but propogation delays may leave you with some downtime). More things than just disks can go bad on a server. 24x7 100% is impossible, but the more redundancy your have the closer you can get to this ideal. Your wallet will be a lot lighter because of your efforts!

ctroyp 01-10-2006 06:15 PM

Thanks for the additional info. It was very informative.

I understand that RAID is not a backup solution, but rather a solution to maintain uptime during disk failure. That is why I would want to use another source for backup. I will have to wait until I get the new server tom implement it completely.

I have tried and tested a dd a couple times and it worked successfully. Like you said, that is not to say it will the next time. But for now, it will get me by until I can make the upgrade...

Thanks again for the informative reply!


All times are GMT -5. The time now is 02:02 AM.