You guys helped me a lot with getting comfortable with some backup strategies. I found a nice 1Tb Seagate USB drive and prepped it with ext4 and a label in fstab.

I was leaning to the cron tar backup but this guy here shows a simple method with rsync and cron to backup his stuff.

Go right to the bottom where the header is:
Backups
And you will see his really simple method. He says that rsync has the advantages of making incremental backups, only the difference once made instead of the entire backup all over again. His method is simple, I do not understand all of the rsync arguments but it looks pretty good as if it would "just work" and be fine, once I tailor it to my own system. Would anyone recommend this and are his rsync commands okay as is?

The author of that page is right.

Rsyncing like this keeps a synchronized copy of your files with minimum effort and minimum use of system resources.

I do my backups for years like this. Especially restoring accidentally deleted or overwritten files is very easy.

I have to recommend that you use 2 backup media alternatingly. One of them might get corrupted during a backup. Or your house might burn down while the media is attached to the computer. Seriously.

Do check if your backup media is mounted. If it is not, you will make the backup on your root file system. Not funny if you expect to have a backup, and not funny if your disk is suddenly filled for 100. (Don't ask how I know this)

This is my backup script:
Note that I call this script from cron. Cron sends an e-mail anything echoed to stdout.

jlinkels

1 members found this post helpful.

EDIT: Bunch of stuff to read below. Bottom line is I made the script and ran it from terminal:
sudo /root/scripts/rsyncbu

It worked well but produced one error:
rsync: readlink_stat("/run/user/1000/gvfs") failed: Permission denied (13)

But I got my backup complete and it did not take that long. Now I put it in cron as root and ran it once in webmin, got this output:
Output from command /root/scripts/rsyncbu ..

rsync: readlink_stat("/run/user/1000/gvfs") failed: Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1052) [sender=3.0.9]
mount point /mnt/backup exists, rsync started

I know that the backup is done and there is essentially nothing new to backup, just wanted to see it run from cron. It will run every day at 5:05 AM as root. Any idea if these errors are serious? Will I get email about them every day?

I asked about how to restore this backup. I know I have a complete copy of my hard drive now and can easily copy from one to the other. But what if one wanted to restore the hard disk? Assuming catastrophic system failure or hard drive died. Can one run rsync in reverse to restore the system or is that asking too much from this script? There would be a LOT of permissions to restore, ownerships, etc. I looked through that previously empty 1 Tb Seagate backup drive, but for the customary lost+found, it was empty. Now it is an *exact* replica of my entire hard drive. With all ownerships and permissions set. I visited /var/log/httpd, could not see, I know that directory is ro for root. What is so amazing is the simplicity of this script. It contained barely enough characters to write two sentences and yet, I now have a total and complete backup of my drive! I am SO happy with it.

There must be a way, if say the hard drive died, to boot to a live CD, and use rsync or even cp -a to restore that entire drive back to a new hard drive. Question is, will it run? I would love to know the proper rsync syntax to restore that backup drive to a new hard drive or overwrite the existing drive in the event of catastrophic system failure from a live Linux CD. There has to be a way. I am very curious about that. Even if it does not allow one to restore the entire hard drive, system and all, it is still a *very* good backup solution! Thanks man, this script so far rocks! jlinkels, you hit one out of the park with this golden nugget bro!

P.S. Thank you, thank you, thank YOU!
===========================================================================================

Holy cow, jlinkels. This is *awesome*! The page author for the backup script is a Psychiatrist who knows a heck of a lot about computers and Linux. He wants to backup his entire hard disk plus an external huge drive. I do not want to do the extra drive, for me it would be just the primary hard disk for right now. I am very much mounted and put the required line into fstab to mount that disk all the time. But checking first would be a BIG plus. It could fail or lose it's power supply and then be gone.

You have some good exclusions in your script. /proc, /sys (Not sure what this dir is for but if you say it is not necessary to backup, I believe you.), /vmware. All good. I can make the changes to tailor it to my own system and would love to try this out tonight. The PC is in the living room now on a 50' Ethernet cable, my old box is on the floor next to my main Windows 7 rig in the bedroom. The Living Room PC will be swapped out with the old Linux machine once the backup completes and I can install the old /home drive and mount that as /mnt/store/. Then I can copy all of my important stuff over locally, rather than pull 265 Gb over the home LAN.

I will copy/paste this into a "backup" script and put it in my scripts directory, make it executable. I have accumulated marvelous scripts for the past oh, 30 years and saving, using them from that scripts dir since 1985 when I was big on Usenet. I learned a LOT from the sysadmins in the Linux newsgroups. As long as you were polite, ask intelligent questions, did not try to get them to "do your homework", or they thought you were a troll, you were welcome to anything you wanted.

I will do this tonight and run it. If it works out okay or not, I will wb tomorrow to mark "Solved" or discuss what, if any issues arise of it.

One BIG Question: I really need this answer. Let's say this just works perfectly for the sake of discussion and disaster happens. My hard drive fails or something. How will I be able to restore my entire hard disk back with this backup? Do you have a restore script to go with this? I am sure it will work but how do you use it go recover your stuff, in the event you have to? Please answer, this is really important. Thank you very much and I will wb tomorrow!

I have created the script and before I even fuss with cron, I will run this by terminal, and see if there is any stdout or email for root. (fingers crossed)

...I have to hold off on testing tonight. This is an older machine that I am refurbishing. For some reason, it is locking up, and will not boot. Not even with previous kernels. I have to debug it, probably hardware. I hope that it is not caps on the mobo.

But please do tell me the restore method while I sort this out. Thank you jlinkels!

EDIT: Got the machine running again. Took out the nvidia card, inspected all the caps on video card and mobo, NO bad caps found. Cleaned video card PCIe terminals with Pink Perl eraser and carefully reinstalled. Took that stupid "quiet" line out of grub.cfg so I can see what is going on and your script is really filling the backup drive with the entire hard drive contents, pretty fast, man!

One stdout error on screen so far:
rsync: readlink_stat("/run/user/1000/gvfs") failed: Permission denied (13)

Not sure what that is about, ran the command as 'sudo /home/paul/scripts/rsyncbu

So far, so good. How do I restore if ever need there be?

Kind of a lot in this thread.

First is ANY backup tends to be better than none. I've had my behind saved so many times with a backup it isn't funny.

Some thoughts.

Errors.
Cron is to be thought of as a user.
Cron rsync can't easily copy entire OS and shouldn't be used for that on a running system. Open files are difficult to rsync sometimes like oracle database and such.
Rsync is one of the least bit transfer means.

Tar has been used for decades.
Tar is slower.
Tar generally is used with a compression of some level to help reduce data. A usb may not need or care if used.


Usb external drives may have a sleep function.

1 members found this post helpful.

Yes, a lot. I tend to talk a lot, something I must work at trying to overcome. I sort of figured this out myself last night but needed confirmation. This backup cannot be used to restore the entire OS and drive to, say, a new hard drive and expect it to work well, if at all. But yes, it certainly did backup everything I have. I can always reinstall the OS and restore my files if that may be the case.

Another manual option is to occasionally reboot the machine to a Ghost, Acronis, or other imaging CD and backup the system hard drive as an image file. This is not a bad idea and I will do that tonight. Such an image would restore the system as it was on that day with all data from up to that day. I have the rsync backup to update data with. Problem with this approach is when left to be done manually, it just does not get done. Even 2 images a year would be better than nothing. Yum update would update the system and data is retained on the backup medium. *Recommendations for hard disk imaging software would be appreciated. I will use what I have for now. Ghost DOS boot CD, Acronis True Image CD, Hiren's BootCD with loads of imaging programs and Linux tools.

For my immediate needs this thread is pretty much solved and I will mark it as such. Unrelated problem for new thread, using old Asus P5GD1, even with updated BIOS will not boot with backup drive plugged in, and Linux will not start until it is plugged in. Saving for new thread. Thanks for the feedback, jefro.

for backup of OS and open files you could use LVM with snapshots. Create a snapshot, mount it, rsync it, delete it.
This and other file or block device based approaches usually don't work for databases, but for DBs you can run a DB dump and backup the dump file with your other files.

1 members found this post helpful.

Last thoughts.

Rsync is great to keep user files and maybe some program files backed up.
It isn't a great image plan unless you boot to some live media. Even then may need some fine tuning.

Tar is a good backup tool booted again to a live media. It is a good tool for user files and some programs.


Generally I like to make a golden or master image of a new install with all updates. Then I try to keep rolling backups of some sort.

jefro, I am trying like heck to make a master image of the new install/hard drive. Both Acronis True Image and Ghost have conniptions about LVM disks. How to you image this sort of a hard drive?

as a offline full backup you could just dd your drives to somewhere else. This will give you a 1to1 block device copy of what's on the HDD/LVM/partition.
if I remember right, Acronis and Ghost internally make use of dd.

to save target storage you could pipe dd output into something like gzip or pbzip2, e.g
or
maybe you want to have a look at dcfldd which is kind of enhanced version of dd, but I never played around with it too much since dd usually is available everywhere and fits all my needs.

1 members found this post helpful.

Thank you DT, now I have the complimentary question. How to restore an image backup made with the above dd commands? I am terrible with syntax on complex issues that I do not fully understand. By the time I "get it right", it will be too late. Just the reverse command lines you would use with the above saves images that I can save to a text file on and off the machine. Thanks again!

Both Acronis and Ghost have a bit by bit method to clone. You have to set it that way if your filesystem can't be read in a file format.

Could be it can't access the drive.

Modern Acronis ought to be able to read lvm I thought.

Instead of a bit by bit clone, consider file based backup. Things like clonezilla (based on partimage and gparted) along with other file type copy may be faster and just as good.

As always, be careful with dd.

That's right. Acronis True Image 2014 on a Linux boot CD did not flat out refuse, it offered a "sector by sector" because it "detected hard drive errors that should be fixed with local disk tools" else use the sector by sector option. I thought there "really were errors" on the disk, a quick 'fsck -avy' would do the trick and that is where LVM became the issue.

Brief history, My Fears of LVM
I was introduced to LVM in Red Hat a LONG time ago. Since it was new, I played with it in webmin and did something to the volume that "completely erased my entire hard disk", or so it seemed. Even with a rescue disk, my hard drive seemed completely empty! I don't remember if this was a system disk or a very important storage disk, so long ago. No way to boot, not even rescue. I made 1 or 2 feeble attempts to restore it without knowing what I was doing and immediately stopped. I *knew* that if I kept messing with it, my chances of recovery would get slimmer and slimmer. After close 10 DAYS to reading and studying everything LVM, lvscan, pvscan, and all the rest, I found out that LVM writes it's ID numbers and settings as the very first few lines of the disk, a buffer. There was a dd command that would let me see it and capture it to a file. I was also warned every time I accessed that disk, the small buffer of info at the beginning of the disk would get pushed further back and out of the buffer! What I captured was the first part of the information, the last part was gone. I used the volume and group ID numbers with a blank LVM "skeleton" file that I could create or existed. I used DD to manually put the ID numbers back into the empty skeleton and peeked into my hard drive, not expecting anything but an empty hard drive and to my astonishment, ALL of my stuff was back! It scared the heck out of me so I try not to mess with it much.

Norton Ghost v11 on a DOS boot CD gave a different message, a bit blurry but I have a photo of the screen. I will type it out because Ghost seems to know all about LVM:

"LVM volumes are cloned as physical partitions. This will probably mean you will need to edit your boot loader and fstab entries to get a bootable system. Proceed with Image File Creation?"

I did not know if that meant it would immediately wreck my current system or that the restored image might not work and I would somehow have to open it with a rescue disk and restore grub and fstab to the way they were? I could save copies of those files and overwrite them or check them as a reference and edit them. I fail to see how creating a system image of the HD would alter fstab or grub anyway so I just backed off.

You are right. For me to remember the proper dd arguments and targets would be dangerous. Clonzilla or something like it would be good. This disc, Hiren's BootCD 15.2, has all of that stuff, free! Scroll down and read some of the Backup Tools section. They even have G4L Ghost 4 Linux 0.34a and I have used these tools with great success! Hiran's Boot CD is essential for Windows and Linux. I am going to try that next. My Acronis is not new enough to work through LVM w/out sector by sector, which you say is alright and I guess it would be. I am popping in Hiran's right now to look at the menu. Can use it in Windows and maybe in Linux, but it *is* a rescue boot CD. Let me get you a screenshot of the Hiren Backup menu. Ok, you can see the attachment. For interested parties, the download link is on a different site, Download Hirens BootCD 15.2. If you get it from torrents, you get the same thing OR the one with all of the commercial stuff included. That menu only shows up in Windows, to get all the good Linux stuff, use it as a boot disc. I am sure you know all this already and I include it only to help other newbies with problems they might have with backup, data or file loss, or might need other tools.

I will give it a try and see if Ghost for Linux will do the job or one of the other varients and let you know. dd is a little scarey for me because I am not familiar enough with it to use it for mission critical stuff and because it WILL write to drives, using dd without understanding it thoroughly or having just the right arguments and targets set is very dangerous.

Attached Thumbnails

 

dd just takes stuff from infile and copies it to outfile. In my examples without compression infile (if) and outfile (of) were just a device and an image file. Working with other programs like gzip you use stin and stout and pipe the data where you want it.
For restore you do it the other way rounf, infile would be your image file, outfile your device. No magic at all.

you might want to google for some dd examples, there aremany howtos out there.

I think you are right, DT. I spent nearly 4 hours late last night trying different backup software from Hiran's disc, kept having issues. And almost 4 hours the other day with Acronis and Ghost, I made ZERO progress in all that time! One of the backup programs I really like but my machine kept freezing up on this gorgeous Linux desktop of the boot disc software and I could not continue. I found some really good dd information sheets. I am not "afraid of it" if I have a good reference and explanation sheet. Man pages are a bit too "dry" for a first timer but make excellent reference pages. Here are some good links:

https://wiki.archlinux.org/index.php/Disk_cloning
http://www.linuxweblog.com/dd-image

Now that not only are good command lines given, but they are explained, I can use dd safely.

This is an OLD machine. I replaced the nVidia card from bad capacitors. I replaced the CPU from 3 GHz 32 bit to 3.6 GHz x64 and am using the same CPU cooler fan with white heat sink paste, not Arctic Silver. The CPU thermal ceiling is 70 C. The machine is running at 65-67 and I get "cannot be reported hardware issues", every few minutes. The ABRT messages give "some idea of what is wrong" and emails root issues. Here is some of it:

:May 19 15:53:34 ohmster mcelog: Please check your system cooling. Performance will be impacted
:May 19 15:53:34 ohmster mcelog: STATUS 3 MCGSTATUS 0
:May 19 15:53:34 ohmster mcelog: MCGCAP 180204 APICID 0 SOCKETID 0
:May 19 15:53:34 ohmster mcelog: CPUID Vendor Intel Family 15 Model 4
:May 19 15:53:34 ohmster mcelog: Hardware event. This is not a software error.
:May 19 15:53:34 ohmster mcelog: MCE 2
:May 19 15:53:34 ohmster mcelog: CPU 1 THERMAL EVENT TSC 2819b8dc8c
:May 19 15:53:34 ohmster mcelog: TIME 1432065195 Tue May 19 15:53:15 2015
:May 19 15:53:34 ohmster mcelog: Processor 1 below trip temperature. Throttling disabled
:May 19 15:53:34 ohmster mcelog: STATUS 2 MCGSTATUS 0

I shut the machine down for now and have to think about this. With these errors every few minutes, I dare not put this machine into service until it runs quietly and smoothly like the machine it will replace. Money issues make me recycle older equipment and this is a good PC. But I do not think the stock 32 bit, 3GHz cooler is sufficient. I will probably need to buy a Cooler Master like I use in my desktop PC but I don't know why, this Linux PC is essentially "doing nothing" right now and this does not make sense! I never saw a PC overheat "just sitting there" unless the fan, heat sink, or both are clogged with dirt and dust. This one has been cleaned. Maybe I cannot use this machine, maybe the new "used" CPU from China is defective. It should not get this hot doing nothing. Even just sitting at run level 3 without X running.

I am stuck. I cannot afford to dump money into a machine that ultimately will not fill it's purpose. Even a stock cooler should not cause overheating if the machine is idling. I have to step back and think about it. Maybe pull the stock Intel cooler, clean it again, and use Arctic Silver and if it still overheats, doing nothing, that is bad. Maybe run it for days from a Linux boot CD and do the same, check CPU temperature in BIOS hardware monitor from time to time. Suggestions welcome. Thanks for the vote of confidence with dd, DT. It does make sense!

dd works, but it takes a long time and wastes a lot of drive space if the source device/partition is huge and only partially filled with non-filesystem data. But it's simple and thorough and predictable and capable of dealing with certain hardware issues.

tar is alright, but I've had issues using it on 32 bit systems if the resulting .tar is > 4GB. It doesn't seem to complain, but if you try to restore from the > 4GB file it fails to be anything bootable. And it gets a bit tedious if you have to tar each individual directory under / and restore them in a particular order to avoid large files. With special considerations for /home and /usr/share/doc/ and others to avoid girth. YMMV

rsync is what I tend to use these days. Although I don't do incremental. And I typically use it to clone a not currently booted to system. Change the fstab, update the grub, and boot the rsync'd copy. When not rsyncing a live system you can omit the long list of --excludes and use simple flags like -aRXv. It's pretty fast and flexible if you can have the down time to not be syncing a live system. You can use it on live systems to, but slower and more complex, plus more potential for undesirable results.

1 members found this post helpful.