gzip directories: 1GB split, with CPU limit? (partial solution figured out)

Ultrus · 12-09-2013, 06:11 PM

Hello,
I'm attempting to backup and compress website on a web server in preparation to be sent to Amazon Simple Storage Service (S3). I've run into the challenge of files needing to be smaller than 5GB before sending to S3 (requiring the split), and when I run the backup, it takes up a LOT of CPU power, and I want to cap that.

Here's what I've got so far.

Original script snippet (CPU sucker, 10GB file being created):

Code:

DirsToBackup[0]='/var/www/vhosts/thewebsite.com'

TmpBackupDir='/home/user/backup'

TodayDate=`date --date="today" +%d-%m-%y`

Today_TmpBackupDir=$TmpBackupDir'/'$TodayDate

## Make the backup directory (Also make it writable)
echo ''
echo 'Making Directory: '$Today_TmpBackupDir
mkdir $Today_TmpBackupDir
chmod 0777 $Today_TmpBackupDir

## GZip the directories and put them into the backups folder
echo ''
for i in "${DirsToBackup[@]}"
do
	filename='dir-'`echo $i | tr '/' '_'`'.tar.gz'
	echo 'Backing up '$i' to '$Today_TmpBackupDir'/'$filename
	tar -czpPf $Today_TmpBackupDir'/'$filename $i
	
done

I'm told this version (untested) may cap the tar process at 5mb per second, also limiting CPU usage:

Code:

DirsToBackup[0]='/var/www/vhosts/thewebsite.com'

TmpBackupDir='/home/user/backup'

TodayDate=`date --date="today" +%d-%m-%y`

Today_TmpBackupDir=$TmpBackupDir'/'$TodayDate

## Make the backup directory (Also make it writable)
echo ''
echo 'Making Directory: '$Today_TmpBackupDir
mkdir $Today_TmpBackupDir
chmod 0777 $Today_TmpBackupDir

## GZip the directories and put them into the backups folder
echo ''
for i in "${DirsToBackup[@]}"
do
	filename='dir-'`echo $i | tr '/' '_'`'.tar.gz'
	echo 'Backing up '$i' to '$Today_TmpBackupDir'/'$filename
	tar -czpPf $Today_TmpBackupDir'/'$filename | pv -L 5m >$i
	
done

Now, how would I combine the following example with the code above?

source: http://stackoverflow.com/questions/1...z-zip-or-bzip2

Code:

# create archives
$ tar cz my_large_file_1 my_large_file_2 | split -b 1024MiB - myfiles_split.tgz_
# uncompress another time
$ cat myfiles_split.tgz_* | tar xz

Would it look something like this?

Code:

DirsToBackup[0]='/var/www/vhosts/thewebsite.com'

TmpBackupDir='/home/user/backup'

TodayDate=`date --date="today" +%d-%m-%y`

Today_TmpBackupDir=$TmpBackupDir'/'$TodayDate

## Make the backup directory (Also make it writable)
echo ''
echo 'Making Directory: '$Today_TmpBackupDir
mkdir $Today_TmpBackupDir
chmod 0777 $Today_TmpBackupDir

## GZip the directories and put them into the backups folder
echo ''
for i in "${DirsToBackup[@]}"
do
	filename='dir-'`echo $i | tr '/' '_'`'.tar.gz'
	echo 'Backing up '$i' to '$Today_TmpBackupDir'/'$filename
	tar -czpPf $Today_TmpBackupDir'/'$filename | pv -L 5m > | split -b 1024MiB - $i_split.tgz_
	
done

Thanks for taking a look! My head is starting to spin.

Best regards,

berndbausch · 12-09-2013, 06:36 PM

Code:

tar -czpPf $Today_TmpBackupDir'/'$filename | pv -L 5m > | split -b 1024MiB - $i_split.tgz_

From a syntax point of view, the ">" in the pv command is wrong. Just remove it.
A bigger problem though: This code creates tar files named $Today...(etc) and pipes the output of tar into pv and then split. I don't think you want to split the tar messages! Instead, create the tar archive on stdout by replacing the $Today... filename with a dash. This should at least be a big step forward, or perhaps the solution.

Thanks for letting me know about the pv command!

Ultrus · 12-09-2013, 08:44 PM

Thanks for the feedback berndbausch.

According to this blog, it looks like I may need to gzip everything first, before splitting it up. If this is true and combined with your great tips, maybe this would look a touch better (untested still, will play tomorrow):

Code:

...

for i in "${DirsToBackup[@]}"
do
	filename='dir-'`echo $i | tr '/' '_'`'.tar.gz'
	echo 'Backing up '$i' to '$Today_TmpBackupDir'/'$filename
	tar -czpPf $Today_TmpBackupDir'/'$filename | pv -L 5m $i
	split -b 1024m $Today_TmpBackupDir'/'$filename part-
	
done

...

unSpawn · 12-10-2013, 01:51 AM

Some comments if I may. tar does hog CPU. You could use 'nice' and 'ionice' (if available) but then the job just takes longer. It's a trade-off. Also compressing files is no requirement: you can basically split any regular file. Also there's other compressors like xz, 7zip and bzip2. Compressing should be done according to file contents: it doesn't make sense to compress files that are already binary formats like MP3 or SWF. Also I'd hash both the original tar ball and the split parts: if after transport an archive doesn't work you only have to efficiently re-transfer the part for which the hash doesn't match. HTH

Ultrus · 12-10-2013, 08:37 AM

Hello unSpawn,
Thanks for your feedback!

For me at this time, compression type is not as important as the number of files being sent to Amazon S3 (for recovery time and # request costs). I have lots of text files, thousands of pre-compressed images.

The length of time required to wrangle the files is not an issue for me, as it will all happen while I sleep and hopefully stay out of the way. I guess that could change as web projects grow.

I appreciate your comment about tar being a CPU hog. I'll definitely look into installing ionice (thanks for sharing!!) and/or making use of nice. Are you aware of xz, 7zip or bzip2 not being a CPU hog?

Catching up on hash now...

Thanks!

unSpawn · 12-10-2013, 01:00 PM

Quote:

Originally Posted by Ultrus

Are you aware of xz, 7zip or bzip2 not being a CPU hog?

No.

Quote:

Originally Posted by Ultrus

Catching up on hash now...

See 'man md5sum' or just

Code:

md5sum /path/file
# or
openssl md5 /path/file
# as in
find /path -type f -print0 | xargs -0 -iX md5sum 'X' > /path/hashes.md5
# or
md5deep -r /path > /path/hashes.md5

berndbausch · 12-10-2013, 05:31 PM

Quote:

Originally Posted by Ultrus

Code:

	tar -czpPf $Today_TmpBackupDir'/'$filename | pv -L 5m $i
	split -b 1024m $Today_TmpBackupDir'/'$filename part-

Let me point out that this code will create the split archive as you intend, but the pv command won't limit IO. You are piping tar's stdout into pv, not the archive. In other words, you can leave out the part after the pipe sign.

Ultrus · 12-11-2013, 10:04 AM

Hmmmm. Is there a way to keep the IO limiting in there somehow? I'm looking into ionice now...

berndbausch · 12-12-2013, 12:51 AM

Quote:

Originally Posted by Ultrus

Hmmmm. Is there a way to keep the IO limiting in there somehow? I'm looking into ionice now...

Something like

Code:

tar -czpPf - $Today_TmpBackupDir'/$filename | pv -L 5m $i | split -b 1024m part-

Note I didn't check this. The main point: in the tar command string, use the dash as the archive name. This writes the archive to stdout, so that it can be piped into pv and ultimately split.
If the $filename is small, though, I doubt that there will be much IO limiting though.