LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 06-18-2010, 11:07 AM   #1
russ00
LQ Newbie
 
Registered: Jun 2010
Location: Australia
Distribution: Puppy Linux 4.12
Posts: 4

Rep: Reputation: 0
Error-correcting filesystem for use with DVDs?


I occasionally need to burn about 1 Gbyte of files to a DVD, and transport them to a friend's house. It's really annoying to discover that some of the data is corrupt when I get there, due to a scratch on the DVD.

I have tried copying the files to the DVD several times, creating a new directory for each set. It usually means at least one of every file is correct, but it's really annoying to have to verify every file is correct with md5sum or similar utility.

What I want is a filesystem that will automatically give me the redundancy I want, even if it means I only get 1 Gbyte of storage space per DVD. When I read the files back, it will automatically fetch multiple copies of the file, and give me the correct data - sort of like the way RAID works, but with all of the data on the DVD.

Does such a filesystem exist? Perhaps I could create three 1Gbyte partitions on my hard drive, mount them with raid, store the files, unmount the partitions, and then copy images of the partitions onto the DVD (mount them later with -o loop,ro options).

Is this feasible, or is there an easier way?
 
Old 06-18-2010, 11:26 AM   #2
David the H.
Bash Guru
 
Registered: Jun 2004
Location: Osaka, Japan
Distribution: Debian sid + kde 3.5 & 4.4
Posts: 6,823

Rep: Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950Reputation: 1950
It's not exactly what you're asking for, but there's an application called dvdisaster that generates an error-recovery (parity?*) file that can be either included on the disk or stored seperately.

(*Edit: I was right. it uses the same kind of Reed-Solomon algorithm used by more general parity systems.)

You can also run cdck on the disk, to determine whether there are any areas on the disk with readability problems.

Last edited by David the H.; 06-18-2010 at 11:45 AM.
 
1 members found this post helpful.
Old 06-18-2010, 09:54 PM   #3
russ00
LQ Newbie
 
Registered: Jun 2010
Location: Australia
Distribution: Puppy Linux 4.12
Posts: 4

Original Poster
Rep: Reputation: 0
Thanks David, your answer is a partial solution to the problem. It solves the data-integrity problem, but not the ease-of use - it can't read files directly from a corrupted DVD, it will only let you create a new (corrected) ISO image.

That will be of some use to me, but then there is the extra step of carrying a copy of DVDisaster with me.

Your link to Parchive is possibly of more use to me. I would still have to carry a copy of the Parchive program (potentially several copies for Mac, PC and Linux), but those could be burned to a directory on the DVD. Parchive also made me remember the RAR archive format, particularly RAR3, which also adds redundancy to the archive, and allows the files to be reconstructed.

Perhaps someone knows of an even better archival format - one that adds error correction AND is mountable? I know gzipped ISO images are mountable, but gzip doesn't add error correction. RAR does both, but the Linux RAR tools can't create archives, they can only open existing archives.

Anyone know of an archive format that (1) adds error correction (2) is mountable and (3) has a creation tool that runs under Linux?

I'm a newbie here - should I start a new thread asking this question?

Last edited by russ00; 06-18-2010 at 09:56 PM. Reason: typo
 
Old 06-18-2010, 10:32 PM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,497

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
My first question would be "why DVD ?". Especially for only a Gig - USB sticks are damn near given away these days. Even in Aus.
Surely both you and your mate have USB ?. You seem to be going to extreme lengths to protect against scratches - do you validate the source file(s) to the same extent ?.
If not, why not ?. Why wouldn't USB (stick or disk) be as valid ?.

To answer the question, Oracle champion zfs for qualities such as this. And it's pretty impressive - but due to license Linus won't allow it in Linux (kernel - there is a fuse option for userspace). I'm currently testing btrfs which does most of what you appear to want. But both you and your mate will need to be on latest kernel levels, and be prepared for the occasional glitch as it's still under heavy development.
That said, I find it fine for day-to-day use as a "normal" (non-RAIDed) filesystem.

Last edited by syg00; 06-19-2010 at 05:06 AM. Reason: Linus - the fellas name is Linus ...
 
Old 06-19-2010, 05:00 AM   #5
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
Here's an idea, get a nice DVD case, a solid one, that way the DVD won't get scratched ... problem solved.

Or another idea: buy a USB stick, one with 4 GB ... problem solved.

I like dvdisaster, but it's mostly for backup purposes, and the ecc is meant to be stored somewhere safe.

Last edited by H_TeXMeX_H; 06-19-2010 at 05:01 AM.
 
Old 06-20-2010, 01:46 AM   #6
russ00
LQ Newbie
 
Registered: Jun 2010
Location: Australia
Distribution: Puppy Linux 4.12
Posts: 4

Original Poster
Rep: Reputation: 0
There are a few reasons why I prefer DVDs to USB sticks:
(1) I have a spindle of blank DVDs already, I don't have any spare USB sticks lying around.
(2) I don't always bring the DVDs back - sometimes they are to give away, e.g. "Can you give me a copy of the video you took at the family reunion".
(3) Everyone knows how to store DVDs, and it's easy to know what is on the DVD, because you can write on it with a pen. Not so easy with USB sticks.
(4) Sometimes files on USB sticks get inadvertently wiped or changed, especially when they are opened with a word-processor "Oops! I hit save!".

The "buy a nice solid DVD case" option will help against scratches, but there are other problems it won't help with. A disk burned and verified on one PC won't always read correctly on another PC, particularly if burn speed was high. I didn't want to put this in my original question, as the question was getting too long already. I also use DVD-RWs a lot, which seem more prone to read errors, particularly after a few burn/erase cycles.

Syg00, thanks for the idea of Btrfs - I haven't kept up with the latest filesystems, and I didn't know it had error recovery. The Wikipedia article on Btrfs only briefly describes the checksumming - I assume this is why you mentioned it - will that allow error correction, or only error detection?

Zfs looks like a step too far in terms of getting it to work on a DVD. I'll look again at my original idea of saving three partition images that were from a RAID array - that seems to be somewhat easier to read, at least with Linux, and (I think) should not need any software at the other end. Anyone ever done this before?

Also, I'm not afraid of using an unusual or rare filesystem. I use a Linux live-cd every day, and I can take it to my friend's house too, and it might encourage them to try Linux!

It looks like this is a harder question than I had anticipated. I thought that surely someone would have developed a filesystem with the features I want, if only for easy data retrieval from backups. Backups weren't what I originally asked for, but I now realise the requirements are almost the same.

Parchive (as linked to by David the H) looks to be my best option, even though it's not a filesystem. If there are no better suggestions in the next two days or so, I'll mark this thread a solved.

Last edited by russ00; 06-20-2010 at 01:53 AM. Reason: Added bit about zfs
 
Old 06-20-2010, 03:34 AM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,497

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
Quote:
Originally Posted by russ00 View Post
Syg00, thanks for the idea of Btrfs - I haven't kept up with the latest filesystems, and I didn't know it had error recovery. The Wikipedia article on Btrfs only briefly describes the checksumming - I assume this is why you mentioned it - will that allow error correction, or only error detection?
No it does error correction, on read (apparently) as well as write - like zfs. I tested by dd'ing zeroes into various files and random locations within the filesystem. Couldn't find any corruption - file comparisons came out the same.
Also tested RAID10 on 4 partitions - then trashed one (entire partition). All the data was still valid. Trying to rebuild the array was less than successful, but that may have been a problem with the test (i.e. me).

Looks pretty solid, but not for production data yet.
 
Old 06-21-2010, 07:06 PM   #8
russ00
LQ Newbie
 
Registered: Jun 2010
Location: Australia
Distribution: Puppy Linux 4.12
Posts: 4

Original Poster
Rep: Reputation: 0
Thanks Syg00, that's good information. I like the "destructive testing" you did - it proves without a doubt that Btrfs does error correction.

You have answered my original question with exactly what I want. Btrfs might not be common at the moment, but it has been in the kernel for over a year, so it should be easy to get hold of.

Now I'm off to see which live-CD distributions support Btrfs. My favourite, Puppy Linux, doesn't mention it, so I'm off to check the Ubuntu lists!

Thanks everyone for your help!
 
  


Reply

Tags
dvd, linux, raid


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Error playing DVDs em21701 Slackware 7 10-24-2009 09:48 PM
LXer: Correcting the Autopano-Sift Hugin Error in 8.10 LXer Syndicated Linux News 0 02-17-2009 05:30 PM
Error w/DVD drive...can't mount DVDs JockVSJock Linux - Hardware 4 10-17-2006 12:59 PM
Error when playing DVDs ValidiusMaximus Linux - Software 1 01-06-2005 10:20 PM
Mplayer and DVDs: This appears to be a libdvdread error Adrohak Linux - Software 4 10-31-2004 06:57 AM


All times are GMT -5. The time now is 08:05 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration