Using Rsync to perform backups
 

While there are many backup threads in this forum none address my concern. I would like a simple way (with a GUI- I don't use the command line because I am the worlds worst typist) to copy (bit for bit) my home directory to an external hard drive.

What I do NOT want to do is what most backup programs insist on doing: put the data into a single directory or file that is not readable except with that program. I do not want to compress, encrypt, process, or otherwise alter the data in any way.

I do not want to save disk space (disk space is almost free now and I have so much I will never fill it). I want the backed up directory to be exactly like and just as readable as the original directory bit for bit. I do not want the backup software skipping protected files, or hidden files, or any other kind of file or data in the directory. I do not want permissions to be an issue. What I am looking for is bit reading not file reading software.

The closest I have found is disk imaging software, but that doesn't work for what I want and it has the annoying tendency to compress files in the image. In my view any kind of compression or alteration of the data into a proprietary format is wasteful, annoying, unnecessary, and extremely and unacceptably risky.

Can anyone advise me? I would be more than willing to pay a premium for such a program since I have been looking for several years without luck.

Disk Imaging for simple "backup"?
Talk about overkill.
There's even a GUI front-end to rsync if that is your preference.

I think this is going to work!
 

Habitual, in my eyes you are a superhero! You are authorized to wear the cape. The software was already on my computer (it came with Mint) but I was unfamiliar with it and it is not listed as a backup utility so I never did anything with it.

Its running now and it looks like it does exactly what I want. WOW, and you gave the answer within minutes of my asking the question.

I owe you!!

ah shucks. ;)

Glad it worked out.

Looked good until I ran a directory comparison
 

Sad to say rsync seemed to be working great but two hours later it stopped with 23 errors and thousands of files were missing. My home user directory has 400,000 items and is 300 gigabytes. My girlfriend's user directory has 100 gigabytes of files. Comparing directories many of the "hidden" directories only had 4096 bytes in them.

Thanks anyway Habitual! I still appreciate knowing about rsync. Maybe I can use it for something else OR maybe I will boot with a Knoppix cd and try using rsync without booting into my home directory (if I can stop Knoppix from using it like the live version of Mint did.)

Habitual had thought using disk imaging software was a bit much but I think that's what I am going to do. You see I use one entire hardrive mounted only as my "home" directory and home is almost 500 gigabytes. That way I can change distributions without affecting home and have done so several times. There are only 18 gigabytes left on it plus it is more than 5 years old.

I need to move everything seamlessly to a new multi-terabyte harddrive. Any attempt to do that while logged in, even as root, has led to incomplete transfers and endless errors. I still also would like to do regular backups that are fully accessible without the backup software and obviously I can't use imaging software for that! I have a whole stack of multi-terabyte disks so space is not an issue.

Maybe I am going about this all wrong?

Run it again. Couple of times.

Turn on logging with
Run it overnight when the computer is inactive?

300 G, you may just run out of memory.
rsync can be a resource hog.

rsync is the answer here IMHO. Thinking about backups is never the wrong idea - imaging has its place, but I use it (very) sparingly.

Thisand this ... makes me think you have errors on the disk. Maybe hardware, maybe fault(s) in the filesystem. You need to investigate every error.
What does dmesg have to say when you get the error messages ?. Have you run SMART tests on the (old) drive ?.
This is the only situation where I will use a bit-for-bit copy - for forensic analysis (of the copy). Note this is not a means of you transitioning to a new disk - all that will do is copy all the errors across. Once you have the copy, you can run fsck and see if there are errors in the filesystem.
ddrescue is another (better) option to try and copy the old disk - it has intelligence to try re-reading (even in reverse order) bad sectors.

Newer versions should be better - although with a large number of files I wouldn't use any of the delete options, just recreate the target dir and run it against a "clean" destination.

rsync is one of the best backup programs out there. I love it. I use it.

It's not new software. And it's used heavily in industry. This means the issues you're having are almost certainly operator error. That's not a dig, it just means you're not a CLI guy. You mentioned this before.

Try this program:
https://github.com/bit-team/backintime

back in time is an rsync based backup tool. I haven't ever tried but it seems very popular, it's actively developed, and it should abstract you a bit from the backend. Hopefully, it will line up the dominos for you :)

I just got back from my trip this weekend and saw all these posts. THANK YOU ALL SO VERY VERY MUCH!!! I know that, like me, the last thing you want to do is spend a lot of time in front of your computer. So taking the time to help the LINUX illiterate is very greatly appreciated!

So I ran fsck both on a live disk logon and as part of Gparted also and got the same results. Doesn't appear to be any errors on the disk but it did fail the smart disk temperature test, running about 152 degrees F. It did say that my disk was, like me, very old! (I used to program in machine language using toggle switches and thought what a great advance it was when I was able to run a teletype through the machine to program it!) I built my first computer in 1970. So like polaris96 suggests (and I am not offended in the least) I got left a tad behind in the technology department.

Also I am running a computer I built using an Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz with 24 gigabytes of memory and a UEFI MSI Motherborard). I doubt I am running out of memory because the CPU and memory monitor doesn't show a whole lot of activity during transfer.

I was using the Luckybackup version of rsync. I will try the backintime tool and also I will try running Habitual's suggested command line if that doesn't work (although that "delete" part scares the crap out of me). I'll report back by the end of the day what if anything worked. I sure don't want to lose 20 years worth of work!

P.S. Just had a thought. The directories are highly organized and the combination of path and file name may approach the 255 character limit for some files. The operating system doesn't seem to mind. Do you suppose rsync doers?

--delete : any files not in the source, delete from the target.

Say yesterday, you rsync'd vacation.jpg and last night you deleted vacation.jpg,
the --delete option will remove vacation.jpg from the target during the next rsync with --delete option.

Without --delete, vacation.jpg remains in the target directory where it is not present in the source any longer.

You should also check for files not owned by yourself with
This may be part of the problem with some "errors".

Wow that opened a can of worms. I have 4 hours of work into this!

I downloaded Backintime but as far as I can tell it does nothing. It says its working but I detected no activity, on the CPU, in memory or on the hard drive.

I did try the command line but the commands do not work.

I ran this line of code with the following errors:
I figured out how to make this work!! NOT. I thought I did but I was wrong yet again. I changed Habitual's command line to this (after about two hours of thought and experiments). Its running and I checked that its writing to the external drive and it is not. It created a directory in /media/q/MyPassport and renamed the drive MyPassport1. Should I use /dev/sdd/ instead?


When I ran the "find" command there were literally thousands of files I did not have permission for. "Find" is a great command, I've got to try to remember it. I deleted all the directories that contained those files except for the ones listed below. Some such as the "rpmdb" may be vestiges of previous flavours of Linux I have had on this computer. I don't understand why ./racket is still there since I deleted it and it no longer appears to be there.

I decided to try luckybackup again since it almost worked before. Maybe with the changes I made it will work this time.

By the way when should I wrap terminal output in "PHP" rather then "code"??

Please post output of
after it's mounted.

OK. I'm going to help you out. Give me an hour or two. :)

I ran the command as requested, Habitual. Here are the results:

I think I figured out what the problem was. I think rsync is working correctly now.

I realized what the period was for in Habitual's command line. Also I understand that the space was not a typo. So I changed it back:


If I am understanding this correctly the dot is telling rysync to copy the current directory. The previous "CD ~ &&" that I removed was to change to the current user's home directory and the "&&" was to add the rsync command. Only the rsync command needed "sudo" to run? I also think I understand now why there is no slash after the first "MyPassport" but there is after the second. I realize now that to make this work all I had to do was add "sudo" after the "&&" and change the target path correctly and it would have worked. Did I understand all this correctly?

At any rate it now appears to be working (and luckybackup no longer works) so I'm using the command line.

If this actually finishes then I will simply create a cron job and run it every Sunday night. I'm sorry. If I had two brain cells left instead of just the one, I could have not wasted everybody's time! At least at work I am still a computer guru because I know Windows very well.:D