Periodic backup of changes on another disk from full backup
I am looking for a backup solution that will enable me to backup only changes to an internal disk after a full backup to an external disk. Also, this should run snapshot backups every day until the next full backup.
Day 1: Full Backup / > External Day 2: Changes to /home after full > internal Day 3: Changes to /home after full > internal etc Currently I am using a simple rsync script for the full backup I have looked and previously used rsnapshot and recently Back in Time, which both allow transparent snapshots on the internal space of timed intervals. Both work great, but rely on full backup to the same location first as far as I know. VG1 root backups here VG2 home I am running Linux, LVM with 2 X VG (one home and other holds root LV and the backup LV). If LVM snapshots can be used in this, then I am open to suggestions. Something similar to https://btrfs.wiki.kernel.org/index....emental_Backup may work if I understand it correctly, but not ready to change to btrfs yet. |
dump/restore (for ext* type filesystems) has levels where 0 means full and 1 means everything since full.
|
Thanks, I didn't know about dump. Only problem is that the files won't remain transparent as in I can't restore an individual files which is the advantage of rsync or rsnapshot style backup. If I lose a file or need to get a file from last week, I don't want to restore the whole partition.
|
Quote:
Code:
restore ivf /name/of/dumpfile |
Fantastic. Thanks I'll look into it. I noticed XFS has this also, which would be preferable for me - if it gets the same result.
|
Since you are already using rsync, you should look into the --link-dest=DIR option.
It works by hardlinks, so one of the requirements is that the destination filesystem must have hardlink capability (as ext3 and ext4 do, and I think even ntfs does). It creates what looks like a full backup; you see a directory with all the files. However, files which have not changed since the last backup are actually stored as hardlinks to the previously saved files, which saves a lot of disk space. Files which were changed or deleted since the last backup can be retrieved by looking in the previous backup directories. Here's how it works: First you create the full backup. If you backup no more than once a day, you could name the backup directory according to the date as in this example: Code:
#!/bin/bash Code:
last_backup=$(ls -1A $backup_dir | tail -1) Additional note added in editing: One of the beautiful things about hardlinking backups in this way is that you can delete old, unneeded backup directories in any order. If you no longer need the first full backup, go ahead and delete it; the incremental backups will still have all the files. Each hardlink is independently associated with the actual file, so as long as at least one hardlink remains, the file is still there. When you delete the last hardlink, the file is gone. |
Quote:
|
Quote:
If the drive is attached to another computer, and accessible via the network, rsync can handle that. It would send the new files and changed files over the network, to be stored on the same filesystem as the full backup. This can be done efficiently by compressing the data stream, and securely with ssh as the transport protocol. I've heard of people doing something like that, creating the full backup locally because it's much faster for a large file system, then taking the drive to their remote office or facility, and then performing incremental backups remotely. If the full backup is off the network, or powered off in a storage closet or a safe, this rsync method won't do what you need. Edit: Your idea to keep the backup at a remote location is an excellent one. It's tragic when the computer and the backup are lost together, as might happen in a fire or theft. |
I have been looking at the xfsdump method and it seems each time it writes a full backup it has to write all the data again. It cannot update just the changes. This would be a fail.
As for the rsync method I have an idea that if I create a LVM snapshot of the current /home and then specify this location as the --link-dest it may fool rsync into thinking that the snapshot was the last backup and then increment the backup from the current home. Not sure on what rsync uses to check the last backup. Haven't tested either of the above yet. |
If you ask for a full backup it writes all the data because that's what you asked for. To do a "differential" dump that writes only the changes you use a different dump level number.
|
Quote:
The dump method does have the advantage of keeping a record of the backup and using that for further backups. What I think your're suggesting is to write diff backups to the external disk and increments to the internal disk. What I understand from this is that the external would grow and grow. Also restoring would be more difficult. Wouldn't I at some point need to run another huge full backup. I only know a little of dump, so I maybe missing something? |
Sorry, I didn't read your original post closely enough. Now that I understand what you want, I have another suggestion. It might not be too hard to write a script to select and backup files that have been created or modified since the last backup, either full or incremental.
Immediately before the full backup, run rsync with the --list-only option and redirect the output to a file on your internal drive: Code:
today=$(date +%F) Code:
/usr/bin/rsync -r /home/ > previous_file_list Then execute the rsync full backup to your external drive. Code:
/usr/bin/rsync -a /home/ $external_backup_dir/$today Code:
/usr/bin/rsync -r /home/ > current_file_list Code:
/usr/bin/rsync -a --files-from=$backup_to-do_list $internal_backup_dir/$today The final step is to update the file list for the next incremental backup: Code:
mv -f current_file_list previous_file_list |
That looks promising. I need to have a look at the compare as I also cannot see an obvious solution at the moment.
What I notice is that using --list-only gives a different and most likely more usable output to -r |
Quote:
Code:
diff previous_file_list current_file_list Quote:
|
There have been some issues this weekend with the system. I lost a drive last week, which prompted the search for a better backup solution. Replaced the disk with a relatively new disk I was using for backups, but now getting Buffer IO errors and system hangs intermittently. I checked the old drive and that is definitely dead. The problem actually looked like a SATA cable for a long time, but today it is looking more like a SATA port.
So, putting the backup solution on hold for a bit until I can sort out the system. Thanks you both for your help so far with this. |
All times are GMT -5. The time now is 05:16 PM. |