Backup script with built in compression while maintaining folder structure

Defested · 09-24-2015, 10:15 PM

I am trying to script a backup of some of my media drives. I would want this backup to maintain the same folder structure as the source, but gzip and files that are larger than 50MB

The method I am using now is something along the lines of

Code:

 rsync -r --progress --update /test/* /test2/
find ./. -size +50M

However, this requires that all of the space of the files be used right away, and then reduced. I am looking for a way for the files to be compressed in transit, while at the same time having rsync only copy the updated files.

unSpawn · 09-25-2015, 05:32 AM

Quote:

Originally Posted by Defested

I am looking for a way for the files to be compressed in transit, while at the same time having rsync only copy the updated files.

That's not how rsync works or is supposed to work. If you still want to use rsync then IMHO your only option is to use a remote file system that performs transparent compression like Btrfs or anything FUSE-based like ZFS or FuseCompress. Or performance / support-wise better: a remote OS that provides ZFS natively.

Defested · 09-25-2015, 09:30 PM

Quote:

Originally Posted by unSpawn

That's not how rsync works or is supposed to work. If you still want to use rsync then IMHO your only option is to use a remote file system that performs transparent compression like Btrfs or anything FUSE-based like ZFS or FuseCompress. Or performance / support-wise better: a remote OS that provides ZFS natively.

If I didn't use rsync, would there be another way?

unSpawn · 09-26-2015, 01:13 AM

Quote:

Originally Posted by Defested

If I didn't use rsync, would there be another way?

Possibly, but not one I know of or could even advise you about.

syg00 · 09-26-2015, 02:16 AM

Have a read of https://wiki.archlinux.org/index.php/Backup_programs - more than you probably want to know.

jmgibson1981 · 09-26-2015, 05:25 PM

Can I suggest incremental backups instead of trying to gzip large files? This uses hard links. Will only backup each file once unless it gets changed. From my manager script. I have this run every hour on a number of directories on my laptop to backup to my server. Wifi both ends takes 10 seconds to do the "backup" on my 2gb /home/$USER folder.

Code:

data_backup() { # versioned backups $1 = name of backed up folder | $2 = /path/to/folder being backed up
	if ping -c 1 "$SERVER" ; then
		if [[ -e "$2" ]] ; then
			if ssh "$USER"@"$SERVER" "[[ -e ${DATABACKUP}/${1}/current ]]" ; then
				rsync -"$RSYNCOPT" --link-dest="$DATABACKUP"/"$1"/current "$2" "$USER"@"$SERVER":"$DATABACKUP"/"$1"/"$1"."$NOW"
				ssh "$USER"@"$SERVER" "rm -f ${DATABACKUP}/${1}/current"
			else
				rsync -"$RSYNCOPT" "$2" "$USER"@"$SERVER":"$DATABACKUP"/"$1"/"$1"."$NOW"
			fi
			ssh "$USER"@"$SERVER" "ln -s ${DATABACKUP}/${1}/${1}.${NOW} ${DATABACKUP}/${1}/current"
			ssh "$USER"@"$SERVER" "find ${DATABACKUP}/${1}/ -type d -mtime +${DATADAYS} -exec rm -fr {} \;"
		fi
	fi
}