Hi,

I have just started experimenting with DVD-RAM's on Linux, to try to make my small/essential backups a little easier/faster.

In doing so, I found out that UDF is not really the way to go on a DVD-RAM in Linux, and decided to try other file systems (speed was terrible, and data corruption occurred too often). Every other file system I've tried works fine (so far), but I would like to hear if anyone has experiences with DVD-RAM?

What file system would give the highest speed reading/writing?

What file system would be safest? (data-loss-wise)

Which file systems would stress the discs the most/least?

Currently, I am testing with ReiserFS, ext2/3 also worked fine. I have no need for using the discs on any other OS, Linux will do just fine..

Would a journalling file system be a good/bad idea? Why?

I'll keep testing and experimenting, and will post my findings, but since I did not find any definitive solution while surfing the web, I hoped I could pick the brains of the forum users a little bit, and see if there is a magic answer out there somewhere...

Sorry you got no help here (yet), did you have any luck on your own? -- Do you have any further results to post?

Well, basically, I'm now using both ReiserFS and ext2/ext3 on different backup discs. I have not been able to test the speed properly (if you have suggestions as to how to do this in a professional manner, they are very welcome). The speed seems acceptable, but I have not been getting consistent results, so I have not posted them.

For now, I've settled with testing stability, and no data have been lost (yet). I'm using rsync for the data transfer itself, to minimize disc stress. The solution seems to be working very good.

I was most of all sceptical after my initial experience with UDF, which was REALLY terrible on Linux. It halted after a short while of data transfer, and went down to a few kB/second, with several seconds of pausing between each file. It also lost a lot of the information transferred, and in my opinion should UDF receive a big fat UNSTABLE warning on Linux.

I'm guessing the same results these file systems produce on a hard disk would apply to this type of medium as well. Next I am going to test whether a ext2 DVD-RAM will be mountable using ext2ifs on windows, if it is, I think I'm going to stick with ext2, to make the backup medium the most compatible. (Even though I really cannot see why I should be restoring any of this data on a MS platform, I'm not enough of a purist to ignore compatibility as a bonus of the solution).

I would like some input on two things still, though;

1. Is there any need for journalling, if rsync checks the CRC anyway, the only result would be less available space on the disc, right?
2. How should/could I test the speed reliably?

Thanks for responding!

and

OK, I'm guessing that the objectives have drifted slighlty. If not, please tell me what is going on.

I think journalling is potentially a bad idea, but it depends on how exactly you are using the disks; its quite possible, in some use cases, that it just doesn't do any good.

At the possible expense of telling you stuff that you already know: Roughly, a journal is a log of changes to be made to the disk, so that if the list of changes is interrupted part way through, the disk can be restored to a consistent state. That's not the same as saying that all of your data is made safe, but, to some degree, the disk is in a consistent state.

So, if you always going to write a complete data set out to the disk, the protection afforded by a journal is minimal. Yes, if the power is cut in the middle of writing out the data to the back set and the original data set is corrupted at the same time, then it may help, to get back some of the data (but inherently, in this situation, it can't guarantee the recovery of all the data) but that really ought to be a very, very rare occurence.

In the more normal case, if the backup dies part way through then you can just re-run the backup to get a good backup set. If you have the situation that the backup can die but the system can restart without you noticing that you have an incomplete data set, then there is a different problem.

Journals must, by their nature, be a rather dynamic data structure. If what you are doing writes the journal to the medium frequently, then the danger is that region gets written often. As media such as optical disks have wear out characteristic, the danger is that in wearing out the area used for the journal prematurely, you are reducing the time until the disk becomes unusable, which isn't an advantage.

Of course, if you only ever write data out to the disk in one total block, including the journal, then you don't wear out the journalling area more quickly; OTOH, if you do this, the journal is, ab definitio, not providing the benefit of recording the status part way through the changes (because that would be something that does not exist) to allow a partial data set to be restored to consistency.

Personally, I don't see what's wrong with using, say, k3b to write out an iso...

I have no experience with your situation but what about FAT32?

IMO preventing data loss during a power outage is best handled by a UPS. Ducking back into my hole now.

The objectives have not drifted that much, but I have not been able to test the speed in a proper manner, so my ways of choosing the "best FS" have been limited. Even though I do not plan to use these discs on any other OS than *nix, it is not important for me to make it unreadable on Windows or other operating systems, so ext2 readability under win or others just seems like something that could be a perk in case I need it sometime.

For instance, if ReiserFS or any other FS is faster and more reliable, I will use that FS, regardless if it is readable on another platform.

My thoughts exactly..

As for your other points, what I am mainly using these backups for, are a bunch of small files that change often, but not many of the files change at once. For instance, I make backups of folders of web pages on my server and my folders of scripts/programs which I write myself, so often only a few files have changed, and writing all of them would be a waste of time. This is why I use rsync. This, however, reflects my personal use of the medium and should not be the main point here, since I really want to find out "Which is the best FS for DVD-RAM on Linux".

My main concern when it comes to data safety is that a file of which I have a good copy on the disc would be overwritten partially or corrupted during a backup, and I would lose the file. Would journalling help in this case?

This might be a dumb question, but if the area storing the journal is worn out, would it not store the journal on another region of the disc?

The thing that made me think that journalling was redundant, was the hardware verification of data on DVD-RAM discs.

Well, since I backup a lot of small files I do not want to waste time burning the whole image every time, and I do not want to use a GUI for the job. If I were to burn out an ISO of all the files every time, I might as well use normal DVD-R discs.

Also, one of the things I would like to do is to put a DVD-RAM in some of the servers I manage, and have a script sync essential files from the server to the disc at intervals.

This would not be a good solution due to the limitations of FAT32, like not being able to preserve file permissions.

Any good ideas as to how I should test the read/write speed to get consistent results?

That was a joke, right? Even M$ was smart enough to abandon FAT32.


What level of consistency are you looking for? IMOPE, the individual file's contents can alter the read/write speed dramatically on any media. i.e., a 2Mb txt file will read/write much faster than a rich htm or high-res jpg of the same size.

Cheers

Retrievil_Knievil,

A journaled filesystem is a bad idea on optical media. The journal causes extra read/write cycles, which slow the overall speed.
You should choose a filesystem that has a physical block size of 32kB, to match the media. Writing smaller blocks is done on a read/modify/write basis, making writes very slow unless write caching is performed either by the drive or operating system.

That only leaves UDF and it defaults to a 2kB block size to suit packet writing on CD-RW media
Its a while since I used UDF with 32kB blocks but it seemed ok on DVD+RW media.

This I know, which is why I need help setting up some sort of standard to test the discs after. I've started looking into bonnie++, but have run into another problem in the mean time....

The DVD-RAM discs I've formatted with ext3 and ReiserFS are now unreadable.....when attempting to mount, I just get a "unknown device" message, and the discs do not show up anywhere, not even in k3b, it just reports "no medium present". !?!

I think this happened after a reboot, but the strange thing is that even though the discs I've previously written to do not show up, I can mount an UDF disc with no problem, and a new disc can be formatted and copied onto with no problems as well...

This makes me wonder if DVD-RAMs are such a good idea after all..

Any ideas, anyone?

Retrievil_Knievil,

what is wrong with ext2? Based on http://en.wikipedia.org/wiki/Ext2, "ext2 is still the filesystem of choice for flash-based storage media (such as SD cards, SSDs, and USB flash drives) since its lack of a journal minimizes the number of writes. Flash devices have only a limited number of write cycles." It must be good for optical based media too.
I'm also dissatisfied with UDF, 2 of 3 of my DVD-RAM/UDF disks went into a state of readonly mode, due to some unknown filesystem error. Because there is no tool to correct the UDF filesystem (udftools contains an empty fsck program), I can do nothing to make them writable again without reformatting the media. I'm about to format one of my disks to ext2 to test it, but if you have additional experience with this filesystem on DVD-RAM, please share with me.

Thanks
brazso

brazso,

ext2 uses a 512B block size on the physical media.
DVD-RAM and the like use a 32kB block size.

To get best performace you need a filesystem that matches its block size to the physical block size on the media or it is in danger of being very slow.

For example, to write a single 32kB physical block on optical media ext2 may do 64 reads and 64 writes, requiring 128 accesses of the disk, needing 128 complete disk revolutions.
Its unlikely to be that bad in reality as caching and read ahead will reduce the number of accesses to the physical media.

In short, ext2 will work but expect it to be slow.

NeddySeagoon,

thanks for your post. Finally I have found what was wrong with my disks. I thought some filesystem error had made them not writable, but somehow they were in write-protected mode. I have restored them into writable state, so for the time being there is no need from my part to choose other filesystem as UDF.
As for the block size, with mkudffs (it is part of the udftools package in linux) I can choose only 1024, 2048 or 4096 bytes per block. It's strange that the optimal value could not be chosen, are you sure that 32kB block size are used physically in DVD-RAM? Anyway how could you format the disc with 32kB value? Do you use cdrwtool command or some other program to format your disks?

Regards,
Brazso

IMO abandoned is a mischaracterization. AFAIK Linux and MS can recognize FAT32. In any event the OP has stated his requirement to maintain file permissions thus eliminating my suggestion.

In my experience a good backup routine requires at least three sets of backups rotated in sequence. Continually overwriting a single DVD-RAM disk IMO would be no better than backing up to a HD. YMMV

brazso,

I cannot find a good reference to cite in support of my 32k block size assertion but I can find a few in support of 2kB.

Certainly CDs use 2kB as that was inherited from the AudioCD format that data CDs were grafted onto.
Audio CDs (for music) actually store more data in the block but for data CDs error recovery information must be added, so you get 2kB of data per block on a CD.