Hi,

I am looking for a tool or method (in Linux) to tar a large amount of data from our backupserver to external 1TB usb harddrives.
Once per month we want to make backups from our backup server (around 3,5TB) to 1TB usb drives. Is there a method, with tar for example, to automatically copy files to a 1TB usb drive and ask for a next drive when the usbdrive is full? Like in the old days with ARJ for example you could archive it to multiple floppy's and would automatically ask for a next volume when the floppy was full.

Now we are using tar to manually create a backup file of each of our servers (we have seperate directories per server on our backupserver) and then copy it to the usb drive. This is taking a very long time as we have to check each time if there is space left to copy the next tar file or that we have to switch to a new usb drive.

Ideal would be that the data on the usb drives is still accessible without using some kind of program to un-extract first.

Hope you understand my problem and have a solution.

Thx

If you have identical size hard drives you can use rsync with -a (archive) to archive the content of one drive to the next. The first time you run rsync -a the entire contents of the source drive will be copied to the backup drive with structure intact. The second time you run rsync -a only those files that have changed since the first rsync will be copied to the backup or archive disk.

If your source disks are 1 gig drives you can use 1 gig drives as backup and in this way gain a 1 to 1 copy of your originals. You can write a script to prompt for the second USB drive or just purchase an equal number of backup external USB drives as your source drives to do your backups. That way you don't need to be present to switch drives.

Alternatively, you could purchase a device such as:

Thermaltake BlacX N0028USU External Hard Drive SATA Enclosure Docking Station 2.5" & 3.5" USB 2.0 which will allow you to quickly swap bare SATA hard drives for backup and then store the bare hard drives off site. The external dock is available in USB or eSATA (external SATA) interface.

or you could install one of these in your server:

KINGWIN KF-1000-BK 3.5" Internal hot swap rack

which will allow you to insert bare removable SATA hard drives into your server for backups.

Thx for the quick reply,

We are already using rsync to backup to our backupserver.
Our backupserver has 16x 1TB in RAID-5 and Linux installed and has 1 large LVM disk of 13TB with XFS filesystem for backup. So i can not simply archive the content from one drive to another.

For the same reason as i mentioned earlier i can not simply copy per disk. I think i really need some clever software solution or script to archive my backupfolder to my external usb drives.

You should look at the dar program. It is similar to tar but is made for archiving to disks. While originally made for CD, DVD, etc. it supports backup slices into the GB, TB, and unbelievably beyond. It can be made to pause after a backup slice is written, allowing you to change USB drives. It also compresses files as they are written and allows individual extraction.

From the dar manpage:

-s, --slice <number>
Size of the slices in bytes. If the number is appended by k (or K), M, G, T,
P E, Z or Y the size is in kilobytes, megabytes, gigabytes, terabytes,
petabytes, exabytes, zettabytes or yottabytes respectively. Example: "20M"
means 20 megabytes, by default, it is the same as giving 20971520 as argument
(see also -aSI and -abinary options). If -s is not present the backup will be
written to a single slice whatever the size of the backup may be (there is
probably some filesystem limitation, thus you might expect problems for file
size over 2 gigabytes, depending on your filesystem, but this is not a limi-
tation of dar).


I also recommend you use eSATA external drives or SATA removable drives. USB is far too slow and obsolete for something this large. If you can fit a RAID cage into your backup server you could backup 3,4,5 removable 1TB disks at a time. If not look into external eSATA arrays with removable trays.

Thx tofino_surfer,
this seems indeed the program that comes pretty close to what i was looking for!

Only drawbacks i can think of is that it stores the files in an archive across the drives so if i want to put back a single file or directory i have to extract it first. Haven't tried that yet, but i think i can overcome this.

Other thing is what happens if one of the disks get corrupt, will i still be able to recover the other undamaged data or is the entire backup broken and can not be recovered ?
This would be crucial to know, because else it would not be a reliable backup methode.

You can extract any single file from the archive as long as you have the index from the first and last archive chunks. As these indexes are critical you could always store multiple copies of the index on different drives.

If you are worried about corruption of your backup drives and you should be, then you should use extra drives for RAID like redundancy of your backup. If you don't have extra backup drives for redundancy then your backup is less reliable then your RAID file server. If you buy two HDs for every dar backup chunk you will have a form of mirror.

Another more radical idea is not to use dar but rather make all of the HDs in your RAID based backup server removable. You could then buy multiple sets of drives and keep one set off site at all times. If you bought a 5 tray RAID cage with 5 extra trays with 10 1GB drives you could have two sets of 4GB capacity ( with RAID 5 ) backup disks. With RAID 5 one disk failure in either set wouldn't cause a problem. With RAID 6 you could have two failures without data loss. You could put a small 50-100MB boot partition on each of the drives so if one failed you could boot successfully from any of them. If you had the money you could make all 16 of your drives removable with multiple hot swap trays and buy 16 extra drives.