LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 10-29-2002, 05:27 AM   #1
system
Member
 
Registered: Dec 2001
Distribution: LFS, RH, Slack
Posts: 104

Rep: Reputation: 15
ECC in tar or something similar?


I'm thinking about getting a cheap tape drive for backup, but tapes and I have a bad history. I would like some way to write to the tape and include redundancy like parity info or something so that I can restore even if I have a read error. I can't find anything like this in the tar info page, though. Are there any tar like progs or else wrappers that can operate on the tar file that will add redundancy to the file? Thanks!
 
Old 10-29-2002, 12:27 PM   #2
akohlsmith
Member
 
Registered: Apr 2002
Distribution: Slackware
Posts: 114

Rep: Reputation: 15
I use afio instead of tar. It compresses each file individually, instead of taking one chunk of data and compressing it as one entity. That way, if your compression tables get hosed, you don't lose everything.

I was able to pick up a nice SCSI-2 DDS4 unit for cheap on ebay. Clean the thing when it says to, keep your tapes in a cool, dry environment like it says to and you should have lots of luck.

Tapes (Travan, DDS, etc.) *already* have all kinds of error detection and correction -- if you're losing files either the tape drive itself is screwed or you're abusing it or the tapes.

I'm sure you know this already, but periodically TEST your backups (i.e. try to restore files) *BEFORE* you need to rely on it to work! This is the same for backing up to tape, cdrom, dvd or another hard drive.
 
Old 10-29-2002, 04:27 PM   #3
system
Member
 
Registered: Dec 2001
Distribution: LFS, RH, Slack
Posts: 104

Original Poster
Rep: Reputation: 15
I don't like relying on hardware ECC alone. Even with, errors do happen. Ideally, I would like hardware ECC, parity in tar, and at least two seperate copies of the backup.

About afio, tar doesn't compress, so tarring to a tape will yeild the same reliability.. If an error occurs, one file gets hosed, but the rest are still ok since there is no compression. I don't really need compression as the data is compressed media to begin with.
 
Old 10-29-2002, 04:43 PM   #4
akohlsmith
Member
 
Registered: Apr 2002
Distribution: Slackware
Posts: 114

Rep: Reputation: 15
You're correct on tar and compression (I compress almost out of instinct now so it's the same thing to me <g>)

What will "software ECC" give you that your two separate copies won't? Two copies in physically separate locations would be very safe, IMO.
 
Old 10-29-2002, 04:47 PM   #5
system
Member
 
Registered: Dec 2001
Distribution: LFS, RH, Slack
Posts: 104

Original Poster
Rep: Reputation: 15
Two tapes + redundancy on each tape is better then two tapes w/o redundancy. It's worth it to me to have both, expecially when using possibly unreliable media (DDS3)
 
Old 10-29-2002, 05:02 PM   #6
akohlsmith
Member
 
Registered: Apr 2002
Distribution: Slackware
Posts: 114

Rep: Reputation: 15
DDS3? Unreliable? My god, compared to what?

I suppose you could find/create a little program that took a byte and spat out a hamming code or other for it. It seems a waste of energy though; if the tape is giving back unreliable data it's better to replace the tape, since you're not guaranteed to get any valid data back (even bits and pieces of the ECC data).

What is wrong with software RAID1 and two physical backups? Or is this an excercise in redundancy?
 
Old 10-29-2002, 05:07 PM   #7
system
Member
 
Registered: Dec 2001
Distribution: LFS, RH, Slack
Posts: 104

Original Poster
Rep: Reputation: 15
compared to DLT IV I've heard. I read reports of DDS3 hardware ECC spitting read errors and not logging them.. I'm extremely paranoid about the data, and currently the data is stuck on a RAID 0 array. I need a backup method before I can build a RAID 5 array, there's too much data and no place to put it.
 
Old 10-29-2002, 05:36 PM   #8
akohlsmith
Member
 
Registered: Apr 2002
Distribution: Slackware
Posts: 114

Rep: Reputation: 15
How much data are you talking about? A 60G drive is cheap these days, and 2 for RAID1 wouldn't be that much more expensive (double, to be exact. <g>).

I've been using DDS4 for years now without issue (and yes I test my backups) -- as far as safety goes, RAID5 isn't all that great / doesn't have that many advantages over RAID1 when it comes to integrity. You still need two drives to go bad to have data loss, and unless you've got a hot spare / hardware RAID card, you won't see anything in terms of better speed / better availability.

We have a DPT (now Adaptec) SCSI UW2 RAID5 array at work with 6 drives and a hot spare. The card wasn't that cheap, but it also does cacheing (with ECC memory) -- no downtime due to the array, anyway.

As far as backup goes at work, we use a DDS3 magazine and do a full backup once a quarter (duplicated to two DVD-Rs which are stored offsite: one in a safe and one at the bank) -- weekly we do an incremental backup from the last week, and then daily we do incrementals from the last day.
 
Old 10-29-2002, 08:34 PM   #9
system
Member
 
Registered: Dec 2001
Distribution: LFS, RH, Slack
Posts: 104

Original Poster
Rep: Reputation: 15
The whole situ: I'm running RAID 0 across two 40Gb partitions to store 60Gb of data. When I rebuild the array I'll buy another 40Gb drive and make it a 3 disk RAID 5 array. My reason for using RAID 5 instead of RAID 1 is capacity. I have two 40Gb partitions that I don't want to scrap to buy new drives. If I run RAID 1, I have 40 Gb space, too little. I could buy two new drives but I'd need to buy really big drives and then I'd not be using my current drives for anything. Alternately, I could go RAID 5. I add one 40Gb drive and get 80Gb of redundant space. That way I have the same amount of space I have now, but with redundancy, and only buy one small (which means cheap) drive.

The only problem with this is that I need to fully back up the 60Gb of data before I initialize the new array. I could buy another very large drive for this function, but then I'd be wasting mony on capacity that would rarely be used. I could not put it in the RAID unless it's 100Gb or greater so that it could hold the 60Gb of data plus the 40Gb RAID partition simultaneously, or else I could not use the drive in the array.

Instead of wasting money on this new drive, I am thinking about getting a tape drive. This way I will be able to back up my RAID array for the conversion, plus I'll be able to keep regular back ups from now on. I've been a little turned off by tape recently, though. I may just use DVD-RAM as I have access to a Mac with one... 9.4Gb per disc, the only pain in the butt is copying the tar files to the mac to burn them manually.

Sigh.. the only thing I wish is that they would add hot drive add code in RAID 5 for Linux software RAID. When I go to add a fourth drive in a year or two, I'll have to restore the whole array from backup. At least I'll have the backups taken care of by then
 
Old 10-30-2002, 12:08 PM   #10
system
Member
 
Registered: Dec 2001
Distribution: LFS, RH, Slack
Posts: 104

Original Poster
Rep: Reputation: 15
Hmm, after looking at drive and media costs, it seems reasonable to just stick with DVD-RW as I already have access to the drive. I guess two copies on that will be fine too as long as I check them at reasonable intervals.

I'd still like ECC in tar, though!

I've looked at some reel to reel drives that use half inch tape.. Too bad they don't have any of those with high density heads and good tape.. You could write the whole internet to one reel!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
how can i decompress this tar.tar file? hmmm sounds new.. tar.tar.. help ;) kublador Linux - Software 14 10-25-2016 02:48 AM
Can't get ecc to load Leftlane Red Hat 0 10-04-2005 08:12 PM
lots of Ram, 2100 DDR ECC sopiaz57 General 3 06-24-2005 04:37 PM
ECC Memory Ztyx Linux - Hardware 2 01-29-2004 05:43 PM
Selection of ECC DDR 266 1024 MB memory tungsten Linux - Hardware 6 08-02-2003 02:02 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:33 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration