LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Hardware
User Name
Password
Linux - Hardware This forum is for Hardware issues.
Having trouble installing a piece of hardware? Want to know if that peripheral is compatible with Linux?

Notices



Reply
 
Search this Thread
Old 04-12-2009, 07:40 PM   #1
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Rep: Reputation: 21
Question DVD-RAM Filesystem - Which is best?


Hi,

I have just started experimenting with DVD-RAM's on Linux, to try to make my small/essential backups a little easier/faster.

In doing so, I found out that UDF is not really the way to go on a DVD-RAM in Linux, and decided to try other file systems (speed was terrible, and data corruption occurred too often). Every other file system I've tried works fine (so far), but I would like to hear if anyone has experiences with DVD-RAM?

What file system would give the highest speed reading/writing?

What file system would be safest? (data-loss-wise)

Which file systems would stress the discs the most/least?

Currently, I am testing with ReiserFS, ext2/3 also worked fine. I have no need for using the discs on any other OS, Linux will do just fine..

Would a journalling file system be a good/bad idea? Why?

I'll keep testing and experimenting, and will post my findings, but since I did not find any definitive solution while surfing the web, I hoped I could pick the brains of the forum users a little bit, and see if there is a magic answer out there somewhere...
 
Old 05-16-2009, 06:40 AM   #2
archtoad6
Senior Member
 
Registered: Oct 2004
Location: Houston, TX (usa)
Distribution: MEPIS, Debian, Knoppix,
Posts: 4,727
Blog Entries: 15

Rep: Reputation: 231Reputation: 231Reputation: 231
Sorry you got no help here (yet), did you have any luck on your own? -- Do you have any further results to post?
 
Old 05-17-2009, 03:30 AM   #3
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
Well, basically, I'm now using both ReiserFS and ext2/ext3 on different backup discs. I have not been able to test the speed properly (if you have suggestions as to how to do this in a professional manner, they are very welcome). The speed seems acceptable, but I have not been getting consistent results, so I have not posted them.

For now, I've settled with testing stability, and no data have been lost (yet). I'm using rsync for the data transfer itself, to minimize disc stress. The solution seems to be working very good.

I was most of all sceptical after my initial experience with UDF, which was REALLY terrible on Linux. It halted after a short while of data transfer, and went down to a few kB/second, with several seconds of pausing between each file. It also lost a lot of the information transferred, and in my opinion should UDF receive a big fat UNSTABLE warning on Linux.

I'm guessing the same results these file systems produce on a hard disk would apply to this type of medium as well. Next I am going to test whether a ext2 DVD-RAM will be mountable using ext2ifs on windows, if it is, I think I'm going to stick with ext2, to make the backup medium the most compatible. (Even though I really cannot see why I should be restoring any of this data on a MS platform, I'm not enough of a purist to ignore compatibility as a bonus of the solution).

I would like some input on two things still, though;

1. Is there any need for journalling, if rsync checks the CRC anyway, the only result would be less available space on the disc, right?
2. How should/could I test the speed reliably?

Thanks for responding!
 
Old 05-17-2009, 05:02 AM   #4
salasi
Senior Member
 
Registered: Jul 2007
Location: Directly above centre of the earth, UK
Distribution: SuSE, plus some hopping
Posts: 3,919

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
Quote:
Originally Posted by Retrievil_Knievil View Post
I have no need for using the discs on any other OS, Linux will do just fine..
and

Quote:
Next I am going to test whether a ext2 DVD-RAM will be mountable using ext2ifs on windows, if it is, I think I'm going to stick with ext2, to make the backup medium the most compatible. (Even though I really cannot see why I should be restoring any of this data on a MS platform, I'm not enough of a purist to ignore compatibility as a bonus of the solution).
OK, I'm guessing that the objectives have drifted slighlty. If not, please tell me what is going on.

Quote:
What file system would be safest? (data-loss-wise)

Which file systems would stress the discs the most/least?

Would a journalling file system be a good/bad idea? Why?
I think journalling is potentially a bad idea, but it depends on how exactly you are using the disks; its quite possible, in some use cases, that it just doesn't do any good.

At the possible expense of telling you stuff that you already know: Roughly, a journal is a log of changes to be made to the disk, so that if the list of changes is interrupted part way through, the disk can be restored to a consistent state. That's not the same as saying that all of your data is made safe, but, to some degree, the disk is in a consistent state.

So, if you always going to write a complete data set out to the disk, the protection afforded by a journal is minimal. Yes, if the power is cut in the middle of writing out the data to the back set and the original data set is corrupted at the same time, then it may help, to get back some of the data (but inherently, in this situation, it can't guarantee the recovery of all the data) but that really ought to be a very, very rare occurence.

In the more normal case, if the backup dies part way through then you can just re-run the backup to get a good backup set. If you have the situation that the backup can die but the system can restart without you noticing that you have an incomplete data set, then there is a different problem.

Journals must, by their nature, be a rather dynamic data structure. If what you are doing writes the journal to the medium frequently, then the danger is that region gets written often. As media such as optical disks have wear out characteristic, the danger is that in wearing out the area used for the journal prematurely, you are reducing the time until the disk becomes unusable, which isn't an advantage.

Of course, if you only ever write data out to the disk in one total block, including the journal, then you don't wear out the journalling area more quickly; OTOH, if you do this, the journal is, ab definitio, not providing the benefit of recording the status part way through the changes (because that would be something that does not exist) to allow a partial data set to be restored to consistency.

Personally, I don't see what's wrong with using, say, k3b to write out an iso...
 
Old 05-19-2009, 02:47 PM   #5
win_to_lin_migrant
Member
 
Registered: Dec 2008
Posts: 125

Rep: Reputation: 16
I have no experience with your situation but what about FAT32?

IMO preventing data loss during a power outage is best handled by a UPS. Ducking back into my hole now.
 
Old 05-24-2009, 03:42 PM   #6
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
Quote:
Originally Posted by salasi View Post

OK, I'm guessing that the objectives have drifted slighlty. If not, please tell me what is going on.
The objectives have not drifted that much, but I have not been able to test the speed in a proper manner, so my ways of choosing the "best FS" have been limited. Even though I do not plan to use these discs on any other OS than *nix, it is not important for me to make it unreadable on Windows or other operating systems, so ext2 readability under win or others just seems like something that could be a perk in case I need it sometime.

For instance, if ReiserFS or any other FS is faster and more reliable, I will use that FS, regardless if it is readable on another platform.

Quote:
Originally Posted by salasi View Post

I think journalling is potentially a bad idea, but it depends on how exactly you are using the disks; its quite possible, in some use cases, that it just doesn't do any good.
My thoughts exactly..

As for your other points, what I am mainly using these backups for, are a bunch of small files that change often, but not many of the files change at once. For instance, I make backups of folders of web pages on my server and my folders of scripts/programs which I write myself, so often only a few files have changed, and writing all of them would be a waste of time. This is why I use rsync. This, however, reflects my personal use of the medium and should not be the main point here, since I really want to find out "Which is the best FS for DVD-RAM on Linux".

My main concern when it comes to data safety is that a file of which I have a good copy on the disc would be overwritten partially or corrupted during a backup, and I would lose the file. Would journalling help in this case?

This might be a dumb question, but if the area storing the journal is worn out, would it not store the journal on another region of the disc?

The thing that made me think that journalling was redundant, was the hardware verification of data on DVD-RAM discs.

Quote:
Originally Posted by salasi View Post

Personally, I don't see what's wrong with using, say, k3b to write out an iso...
Well, since I backup a lot of small files I do not want to waste time burning the whole image every time, and I do not want to use a GUI for the job. If I were to burn out an ISO of all the files every time, I might as well use normal DVD-R discs.

Also, one of the things I would like to do is to put a DVD-RAM in some of the servers I manage, and have a script sync essential files from the server to the disc at intervals.

Quote:
Originally Posted by win_to_lin_migrant

I have no experience with your situation but what about FAT32?
This would not be a good solution due to the limitations of FAT32, like not being able to preserve file permissions.

Any good ideas as to how I should test the read/write speed to get consistent results?
 
Old 05-24-2009, 04:41 PM   #7
DragonSlayer48DX
Registered User
 
Registered: Dec 2006
Posts: 1,454
Blog Entries: 1

Rep: Reputation: 74
Quote:
Originally Posted by win_to_lin_migrant View Post
I have no experience with your situation but what about FAT32?
That was a joke, right? Even M$ was smart enough to abandon FAT32.


Quote:
Originally Posted by Retrievil_Knievil View Post
Any good ideas as to how I should test the read/write speed to get consistent results?
What level of consistency are you looking for? IMOPE, the individual file's contents can alter the read/write speed dramatically on any media. i.e., a 2Mb txt file will read/write much faster than a rich htm or high-res jpg of the same size.

Cheers
 
Old 05-25-2009, 03:33 PM   #8
NeddySeagoon
Gentoo support team
 
Registered: May 2009
Location: 56N 3W
Distribution: Gentoo
Posts: 178

Rep: Reputation: 40
Retrievil_Knievil,

A journaled filesystem is a bad idea on optical media. The journal causes extra read/write cycles, which slow the overall speed.
You should choose a filesystem that has a physical block size of 32kB, to match the media. Writing smaller blocks is done on a read/modify/write basis, making writes very slow unless write caching is performed either by the drive or operating system.

That only leaves UDF and it defaults to a 2kB block size to suit packet writing on CD-RW media
Its a while since I used UDF with 32kB blocks but it seemed ok on DVD+RW media.
 
Old 05-25-2009, 04:11 PM   #9
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
Quote:
Originally Posted by dragonslayer48dx View Post
What level of consistency are you looking for? IMOPE, the individual file's contents can alter the read/write speed dramatically on any media. i.e., a 2Mb txt file will read/write much faster than a rich htm or high-res jpg of the same size.
Cheers
This I know, which is why I need help setting up some sort of standard to test the discs after. I've started looking into bonnie++, but have run into another problem in the mean time....

The DVD-RAM discs I've formatted with ext3 and ReiserFS are now unreadable.....when attempting to mount, I just get a "unknown device" message, and the discs do not show up anywhere, not even in k3b, it just reports "no medium present". !?!

I think this happened after a reboot, but the strange thing is that even though the discs I've previously written to do not show up, I can mount an UDF disc with no problem, and a new disc can be formatted and copied onto with no problems as well...

This makes me wonder if DVD-RAMs are such a good idea after all..

Any ideas, anyone?
 
Old 05-29-2009, 04:08 AM   #10
brazso
LQ Newbie
 
Registered: May 2009
Posts: 3

Rep: Reputation: 0
Retrievil_Knievil,

what is wrong with ext2? Based on http://en.wikipedia.org/wiki/Ext2, "ext2 is still the filesystem of choice for flash-based storage media (such as SD cards, SSDs, and USB flash drives) since its lack of a journal minimizes the number of writes. Flash devices have only a limited number of write cycles." It must be good for optical based media too.
I'm also dissatisfied with UDF, 2 of 3 of my DVD-RAM/UDF disks went into a state of readonly mode, due to some unknown filesystem error. Because there is no tool to correct the UDF filesystem (udftools contains an empty fsck program), I can do nothing to make them writable again without reformatting the media. I'm about to format one of my disks to ext2 to test it, but if you have additional experience with this filesystem on DVD-RAM, please share with me.

Thanks
brazso
 
Old 06-02-2009, 03:41 PM   #11
NeddySeagoon
Gentoo support team
 
Registered: May 2009
Location: 56N 3W
Distribution: Gentoo
Posts: 178

Rep: Reputation: 40
brazso,

ext2 uses a 512B block size on the physical media.
DVD-RAM and the like use a 32kB block size.

To get best performace you need a filesystem that matches its block size to the physical block size on the media or it is in danger of being very slow.

For example, to write a single 32kB physical block on optical media ext2 may do 64 reads and 64 writes, requiring 128 accesses of the disk, needing 128 complete disk revolutions.
Its unlikely to be that bad in reality as caching and read ahead will reduce the number of accesses to the physical media.

In short, ext2 will work but expect it to be slow.
 
Old 06-03-2009, 03:20 AM   #12
brazso
LQ Newbie
 
Registered: May 2009
Posts: 3

Rep: Reputation: 0
NeddySeagoon,

thanks for your post. Finally I have found what was wrong with my disks. I thought some filesystem error had made them not writable, but somehow they were in write-protected mode. I have restored them into writable state, so for the time being there is no need from my part to choose other filesystem as UDF.
As for the block size, with mkudffs (it is part of the udftools package in linux) I can choose only 1024, 2048 or 4096 bytes per block. It's strange that the optimal value could not be chosen, are you sure that 32kB block size are used physically in DVD-RAM? Anyway how could you format the disc with 32kB value? Do you use cdrwtool command or some other program to format your disks?

Regards,
Brazso

Last edited by brazso; 06-03-2009 at 03:21 AM.
 
Old 06-03-2009, 11:23 AM   #13
win_to_lin_migrant
Member
 
Registered: Dec 2008
Posts: 125

Rep: Reputation: 16
Quote:
Originally Posted by dragonslayer48dx View Post
That was a joke, right? Even M$ was smart enough to abandon FAT32.
IMO abandoned is a mischaracterization. AFAIK Linux and MS can recognize FAT32. In any event the OP has stated his requirement to maintain file permissions thus eliminating my suggestion.
 
Old 06-03-2009, 11:39 AM   #14
win_to_lin_migrant
Member
 
Registered: Dec 2008
Posts: 125

Rep: Reputation: 16
Quote:
Originally Posted by Retrievil_Knievil View Post
This makes me wonder if DVD-RAMs are such a good idea after all..

Any ideas, anyone?
In my experience a good backup routine requires at least three sets of backups rotated in sequence. Continually overwriting a single DVD-RAM disk IMO would be no better than backing up to a HD. YMMV
 
Old 06-04-2009, 02:28 PM   #15
NeddySeagoon
Gentoo support team
 
Registered: May 2009
Location: 56N 3W
Distribution: Gentoo
Posts: 178

Rep: Reputation: 40
brazso,

I cannot find a good reference to cite in support of my 32k block size assertion but I can find a few in support of 2kB.

Certainly CDs use 2kB as that was inherited from the AudioCD format that data CDs were grafted onto.
Audio CDs (for music) actually store more data in the block but for data CDs error recovery information must be added, so you get 2kB of data per block on a CD.
 
  


Reply

Tags
backup, dvdram, filesystems, udf


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Throwing a Filesystem (or part of it) Into RAM. Any Idea on How to Do This? LinuxGeek Linux - Software 1 01-26-2006 02:10 PM
LG DVD+-RW/RAM won't mount after writing DVD (growisofs/k3b) jaws_tas Linux - Hardware 1 03-25-2005 02:52 AM
How to play dvd s in regular user account? DVD-RAM How-To? Outabux Debian 1 07-08-2004 08:51 PM
Creating a Virtual FileSystem on RAM (not VFS!!)? shrey_j Programming 0 07-06-2004 04:22 AM
Ext2 Filesystem in RAM Fernando Linux - General 0 03-21-2002 03:32 AM


All times are GMT -5. The time now is 05:35 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration