I'm looking for some advice. We have been making nightly backups for the last several years on Ultrium LTO 2 tape drives, with the tapes stored off-site. Unfortunately, these tapes no longer provide enough space for our backups. We are looking for something stable that can handle at least 400 GB of uncompressed data and can complete the transfer in 10 hours at most. We are currently considering finding a new tape drive or using external hard drives. Are there any solutions we are overlooking? Does anyone have any input on either of these options? Are external hard drives stable enough to use as a reliable backup solution, or is that too risky?

IMO, No more or less shaky than using Magnetic Tape for backups...

• One stray magnetic Field, you have no data
• If the read head crumples the tape, you have no data
• If something spills on it, you have no data
• If dust gets in the tape, then on the write head, you have no data
• If the thin magnetic tape breaks, you have no data

(Obivously in some circumstances, *some* data will survive)

Plus, It takes days to pull stuff off of tapes.

Backing up to multiple offsite arrays is nice. When able, I back up to Bluray disks, or for short term storage, USB hard drives.

One option that might suit your needs is Crashplan.
http://www.crashplan.com/
With Crashplan you can do onsite backup and offsite backup for free.
Additionally there are paid options that will give you up to unlimited online storage for a relatively low price.
I use Crashplan for my personal backup solution as it is cross platform and I work in both the Linux and Windows world.

Are you backing up a network of business computers or your own personal computer? In a work environment, backup policies and requirements affect your choices. You may need to be able to retrieve a file from any point in time going back months. In other situations, the existence of backup tapes going back forever can be a liability (say, if you are hit with a freedom of information request demanding all back emails to or from . . . ).

I run tape libraries with around 6 weeks worth of daily tapes. The current versions of LTO are very fast and have lots of space. Your LTO2 is three generations back (I'm sure you know that ). If you upgrade your tape drive, you should also consider upgrading your backup server. Poor performance of LTO4 or 5 can be because the computer is unable to keep up with them.

400G on a daily basis is obviously not going to work going out to any cloud service. Using disk drives can be a simple solution for personal data, but it gets out of hand pretty quickly in a business network environment. You do want multiple levels of protection. So, having mirrored boot drives on critical servers, or raid (6 or z2 for very large arrays) for critical data, is also important. Having rolling snapshots (with zfs, with rsync, or with something like NetApp) can be another useful protection for short cycle file recovery. And, of course, if you are running databases, you need to account for proper dumping of those for backup.

In whatever case you choose, consider redundancy. Using disk drives? Make sure you have enough so that if disaster hits, you are covered. Be extra paranoid. That's why I like tapes. If a tape is messed up, I know I have many other copies on other tapes. If a tape drive is messed up, repair or replace it and I can still read my tapes. I may lose the copy from one day, but I've got lots of others. Modern tapes are much more reliable than the older tapes. I've only lost one tape across four tape libraries in the last 4 years. That's with over a hundred tapes being cycled constantly.

1 members found this post helpful.

Thanks for all the good info everyone. It looks like I haven't given enough information for some of you though, so here's a bit more. We are backing up our production data, sans financial information, but this is something of a last resort backup. Our production server has a hot spare with a full backup (both housed at a separate site), which is then backed up to our backup server, which has a Raid array. That server is the one from which we are making the tape backups, or potentially backing up to a portable HDD, to be stored off-site. Our current arrangement is to have 3 tapes that we rotate through. We aren't required to comply with any standards for our backup tapes (at least for the time being), but we want to have a copy of our data at a third location just in case some catastrophe should strike.
The contents of our backups is mainly image files and PDFs, as well as several different database dumps. These dumps are all archived on our backup server, but for our purposes, only the most recent is stored on the removable media.

Using external disks only is for me far away from reliable. Also optical disc is no real solution.

LTO4/5 should solve your problem, as mentioned by choogendyk. In a small library for media managent you can transfer > 100MB/s ~ 320GB/h to 400GB uncompressed media (hardware compressed ~ 680GB) on a GBit LAN without any problem. I'm not sure what size LTO5 is, actually i did'nt use it.

Using staging disk on the backup server provides flexibility to put smaller files (eg your pdf files) into larger junks and transfer the larger junks to tape to get better performance. Also for long term and revisionable backup WORM (write once read many) media is support.

Snapshots are great for getting a temporary state of a storage area, but not really for long term backups because of cost.

Inspite of all hardware and performance, but an intelligent, documented and tested backup and recovery process should be implemented to assure the data is correctly recoverable. So i think it should be no problem to backup your data within 3h. Restore can be done within the same time.

@szboardstretcher: yes, tape has limitations, but from my point performance is not the problem. Also if you ever have tried to carry TBytes of data to a external location you will find the disk storage unit somewhat unhandy

Wikipedia is often a handy reference for this kind of thing: http://en.wikipedia.org/wiki/Linear_...en#Generations.

I understand the cost of adding more tapes. Although I recommend lots of tapes (say, daily tapes to cover 6 weeks or more), I have one department that can never come up with money. It took 3 years for them to come up with a tape drive. They only came up with a handful of tapes. So, I run daily incrementals to disk and flush it all out to tape with the fulls on the weekend. That way, I cover them for weeks instead of days.

As I mentioned previously, with all the legal stuff nowadays long term backups can be a liability. However, I was working for a software developer in the 90's. The head of the company came to me looking for a demo movie that he had the year before for a trade show. He wasn't sure of the name or which computer it was on, but had a basic description. The trade show was coming up again, and he wanted this demo movie to go along with the current materials. Because we kept backup tapes forever, never overwriting them, I was able to go back to the year before and retrieve the demo movie. Those were the days when hard drives were measured in megabytes (http://upload.wikimedia.org/wikipedi...r_time.svg.png) and the tapes (DDS-2) were 4GB.

The cost of the tapes isn't really the reason we only use 3 tapes- we simply don't need any more than that. With 3 tapes, we can guarantee that we have a full backup off-site at all times, which is the sole reason we keep our tape backups. That covers all the information we need, and more tapes just seems superfluous.

Or perhaps insufficiently paranoid?

One of the tricks of a sysadmin is to learn to imagine all kinds of failure scenarios. Suppose a fire at your data center and, coincidentally, the *one* tape that is currently off site is corrupt. The disaster results in all your eggs being in that one basket. Or, suppose that a critical file is corrupted and you don't notice it until the backups have already been rotated. Gee, it's corrupt here, and it's also corrupt on the off site backup -- we just over wrote the last good copy. Or, you failed to use a cleaning tape over a very long time, the drive has started writing glitches, and you also failed to test your backup tapes for recovery recently (routine complacency). You get a drive failure and then discover you can't read any of your tapes.

After spending time imagining all sorts of possible failure scenarios, then you plan a system that balances risk and cost. If the effort or cost is too high, and/or you're willing to live with the risk (or your boss is), then choose the less costly system/design. In some cases, you can throw on extra levels of protection at virtually no cost just to assuage your paranoia. For example, ZFS or rsync snapshots can be almost cost free (some of your time in implementation and relatively little disk space).

I have spent a bit of time considering such situations. I'm sure I've overlooked things, but even in the scenarios you have mentioned, if the datacenter and the off-site tapes are both corrupt, I still have a production server and it's backup off-site that have a full copy of all necessary data. The way I see it, our tape/disk backup solution I am considering is effectively our 4th or 5th level of redundancy (depending on the situation).

Cool. Sounds like you may be covered then.