LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 01-10-2006, 12:36 PM   #1
ctroyp
Member
 
Registered: Sep 2005
Distribution: Fedora Core 4 / CentOS
Posts: 92

Rep: Reputation: 15
Backup script to write to file?


Hello, I have already created a backup script that dd's hda to hdb. The script is placed in /etc/cron.daily. Within the script, I write the current date and time to a backup log, run the dd command, then I write another date and time to the same log to show how long the backup took.

I have two questions:
1)When running the dd command manuall in the shell, it returns the results of how much information was read from hda and how much information was written to hdb. I would like to write these results to the same log file along with the date and times that are already being written. I am unsure how to do this... Could someone help?

2)I would also like to calculate the elapsed time using the two dates and write it to the log file as well. This would allow me to monitor the time it takes to run the backup as the data grows. Any idea on how to create the calculation? I looked this up on google, but couldn't find what I was looking for.

Thanks in advance for any help offered.
 
Old 01-10-2006, 12:43 PM   #2
ask.jackal
LQ Newbie
 
Registered: Jan 2006
Location: India
Distribution: Suse
Posts: 12

Rep: Reputation: 0
1. Redirect standard error to a file like below.

dd if=/dev/null of=/tmp/test 2>logfile

2. Not sure of any easy method. You may have to use ur prgmming skills !!
 
Old 01-10-2006, 12:48 PM   #3
ctroyp
Member
 
Registered: Sep 2005
Distribution: Fedora Core 4 / CentOS
Posts: 92

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by ask.jackal
1. Redirect standard error to a file like below.

dd if=/dev/null of=/tmp/test 2>logfile
Excellent! You made that easy...

Quote:
Originally Posted by ask.jackal
2. Not sure of any easy method. You may have to use ur prgmming skills !!
That's the problem, being a newbie, I have been pulling my hair out. I'm not bald yet so I guess I'll keep trying.
 
Old 01-10-2006, 12:53 PM   #4
ctroyp
Member
 
Registered: Sep 2005
Distribution: Fedora Core 4 / CentOS
Posts: 92

Original Poster
Rep: Reputation: 15
When using the redirect, will it overwrite the file? I want to append the log file so should I use this?
Code:
dd if=/dev/null of=/tmp/test 2>>logfile
 
Old 01-10-2006, 01:00 PM   #5
ctroyp
Member
 
Registered: Sep 2005
Distribution: Fedora Core 4 / CentOS
Posts: 92

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by ctroyp
When using the redirect, will it overwrite the file? I want to append the log file so should I use this?
Code:
dd if=/dev/null of=/tmp/test 2>>logfile
Disregard, I found the answer--'yes'.
 
Old 01-10-2006, 02:55 PM   #6
jrdioko
Member
 
Registered: Oct 2002
Distribution: Debian 6.0.2 (squeeze)
Posts: 944

Rep: Reputation: 30
And if you want all the output that would normally be spit out in the terminal, not just the error stream, use ">>" instead of "2>>".
 
Old 01-10-2006, 03:07 PM   #7
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD
Posts: 1,996

Rep: Reputation: 302Reputation: 302Reputation: 302Reputation: 302
Quote:
Originally Posted by jrdioko
And if you want all the output that would normally be spit out in the terminal, not just the error stream, use ">>" instead of "2>>".
That would get you stdout, but not stderr. To get both:
Code:
dd if=/dev/null of=/tmp/test >>logfile 2>&1
To SEE it on your screen, and LOG it to a file at the same time:
Code:
dd if=/dev/null of=/tmp/test 2>&1 | tee -a logfile
 
Old 01-10-2006, 03:23 PM   #8
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD
Posts: 1,996

Rep: Reputation: 302Reputation: 302Reputation: 302Reputation: 302
Aside from your main question of how to log program output, using dd to backup a filesystem while that filesystem is mounted and actively being used will not get you a good clean backup copy. So when you're dd'ing an entire harddisk as in your example, you don't want to be running out of any OS installed on that disk. Things would be changing on disk WHILE you're dd'ing, resulting in inconsistancies at minimum, something totally unuseable worst case. To dd a disk, boot Knoppix or some other live CD. To dd a partition, first unmount the filesystem using that partition, or at least make sure it's a stable not-normally-changing filesystem if you still feel adventurous. For something as important as a backup, I wouldn't recommend going the adventurous route. You might be surprized later when you actually need that backup!
 
Old 01-10-2006, 03:26 PM   #9
jrdioko
Member
 
Registered: Oct 2002
Distribution: Debian 6.0.2 (squeeze)
Posts: 944

Rep: Reputation: 30
Quote:
Originally Posted by haertig
That would get you stdout, but not stderr.
Interesting, never did read up on my output streams in any depth. I stand corrected.
 
Old 01-10-2006, 03:53 PM   #10
ctroyp
Member
 
Registered: Sep 2005
Distribution: Fedora Core 4 / CentOS
Posts: 92

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by haertig
Aside from your main question of how to log program output, using dd to backup a filesystem while that filesystem is mounted and actively being used will not get you a good clean backup copy. So when you're dd'ing an entire harddisk as in your example, you don't want to be running out of any OS installed on that disk. Things would be changing on disk WHILE you're dd'ing, resulting in inconsistancies at minimum, something totally unuseable worst case. To dd a disk, boot Knoppix or some other live CD. To dd a partition, first unmount the filesystem using that partition, or at least make sure it's a stable not-normally-changing filesystem if you still feel adventurous. For something as important as a backup, I wouldn't recommend going the adventurous route. You might be surprized later when you actually need that backup!
I am going after a new server w/RAID, backup tapes, etc. I am doing this as a temporary means to backup (what I can get) my system. I am unaware of another way to backup files on a webserver that is constantly working. Plus, I cannot take the server down to do a dd one time. Files change every minute and I have to backup daily at minimum.

Thanks for the feedback and feel free to give me any other ideas.

Also, does anyone have an idea on the elapsed time calculation?
 
Old 01-10-2006, 04:02 PM   #11
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD
Posts: 1,996

Rep: Reputation: 302Reputation: 302Reputation: 302Reputation: 302
Quote:
Originally Posted by ctroyp
Thanks for the feedback and feel free to give me any other ideas.
Use "rsync" to do these type of live backups. You can get fancy with excludes and all kinds of other stuff, but basically it's just something like:
Code:
rsync -va /usr /other_disk/other_dir
If files go changing out from under rsync, it will "do the right thing" (TM) ;-)
 
Old 01-10-2006, 05:04 PM   #12
ctroyp
Member
 
Registered: Sep 2005
Distribution: Fedora Core 4 / CentOS
Posts: 92

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by haertig
Use "rsync" to do these type of live backups. You can get fancy with excludes and all kinds of other stuff, but basically it's just something like:
Code:
rsync -va /usr /other_disk/other_dir
If files go changing out from under rsync, it will "do the right thing" (TM) ;-)
Okay, thanks a lot. I'll check into it. One question though...will the drive rsync'd to be a bootable drive?
 
Old 01-10-2006, 06:07 PM   #13
haertig
Senior Member
 
Registered: Nov 2004
Distribution: Debian, Ubuntu, LinuxMint, Slackware, SysrescueCD
Posts: 1,996

Rep: Reputation: 302Reputation: 302Reputation: 302Reputation: 302
rsync copies files. It does not create disk images. So no, your backup would not be bootable. But I think you would be in better shape to restore from rsync backups rather than with only the live dd's you have now. Do both - your live dd just may work, by shear dumb luck. To do a bare metal recovery you would typically need to first create a minimally bootable installation - either from your dd disk image (if it worked!) or by installing the minimal OS from scratch. But then you could bring this recovery system back up to full capabilities from your rsync backups.

Don't consider RAID a backup. It's a fallback, not a backup. You can recover an accidently deleted file from backups, but not from RAID. If a disk goes bad, a set of backups will not keep your server up and running during the fault, but RAID might. Two different things. I'm sure you know this already. But you'd be surprized how many people think RAID is a substitute for backups. If your webserver really must stay up 24x7 you should investigate using multiple computers with some type of IP takeover scheme (DNS changes can do some of this, but propogation delays may leave you with some downtime). More things than just disks can go bad on a server. 24x7 100% is impossible, but the more redundancy your have the closer you can get to this ideal. Your wallet will be a lot lighter because of your efforts!
 
Old 01-10-2006, 06:15 PM   #14
ctroyp
Member
 
Registered: Sep 2005
Distribution: Fedora Core 4 / CentOS
Posts: 92

Original Poster
Rep: Reputation: 15
Thanks for the additional info. It was very informative.

I understand that RAID is not a backup solution, but rather a solution to maintain uptime during disk failure. That is why I would want to use another source for backup. I will have to wait until I get the new server tom implement it completely.

I have tried and tested a dd a couple times and it worked successfully. Like you said, that is not to say it will the next time. But for now, it will get me by until I can make the upgrade...

Thanks again for the informative reply!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
script to analyze system then write a .config file for kernel compiling jmannik Linux - Newbie 7 12-19-2005 02:15 PM
How to write a shell script to download a file via FTP? guarriman Linux - General 4 12-21-2004 09:31 AM
Red Hat 9 server can't write PHP script, problem in httpd.conf file? x_menno_x Linux - Newbie 27 06-29-2004 04:44 PM
php script can not write text file lemotion Linux - Newbie 5 04-20-2004 10:14 PM
Need script to dump database, ftp file to backup server glock19 Linux - Networking 17 05-12-2003 05:56 PM


All times are GMT -5. The time now is 12:19 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration