Good bye data, good bye 8 years worth of stuff.
 

Well, I have now just experienced the worst data loss ever. I have (or had I should say) an external drive, with all of my pictures (vacation), movies, and mp3 collection for the past oh I don't know, 8 fscking years!! (MP3: 113GB). Anyways, I kept hearing about my friend's issue with recordable media, and so thats why about a year ago, I bought an external hd, to keep it all on that. Well, until today, everything was great, until I turned it on to retrieve data off of it. As soon as I did, I was greeted by a very lovely ear piercing chainsaw like sound, I have no way of getting at the data now. Approximately, 260, out of 300GB, GONE!:eek: :mad: :cry: :cry: :cry: :cry:

I would have also archived to recordable media, but I just don't have any right now, so yes I'm very well much fscked. I went online, and looked at recovery services, fsck they don't come cheap, $750 to $2000,:eek: I don't have that kind of money. At this point I am just asking myself, and wondering, if ever I do buy another external, what is the best reliable recordable media to buy? If I have to, next time I should make extra copies, and keep it in a dry, cool area, in avault somewhere, not just an external hd.

I need a stiff drink, where's that bottle of Jack Daniels? :cry: :cry: :cry:

the morale to the story, keep everything backed up

Sorry if I seem to be lashing out at you, but isn't that what an external drive is for!? :scratch:

If it's your only copy it's just archived, not backed up.

This makes me wondere what condition all my CDRs are. The CMC magnetics manufactured ones are probably completely deteriorated. Most of the riteks probably have errors. I better check the maxells and yudens to make sure at least those are fine.

Ironically I had most of my pictures and MP3s on CDs, but these CDs were relatively old, not scratched, good decent brands, and I had a hard time getting data off of some of them. Which makes me wonder, is there really such a thing as an archival quality CD/DVD-R? Or Maybe I should send my data to some company that can make me stamped discs rather than burned disks?

Luckily for most if not all the pictures, I have other friends and family that I can get copies from, but as far as the music and movies, gone, and misc. data over time that I held on to, gone. I specifically bought the damn thing so I wouldn't worry about 'laser rot' on recordable media, and it's not like I used the thing every day, and I never left it on when not in use either. I just don't get it.

I feel sorry for you Jeebizz. I also have two external, USB-HD's (one Samsung and one Western Digital) and I can feel they won't last long. Both get extremely hot when in use. The WD one can sound weird sometimes when spinning.

In Windows (unsure how it will behave in Linux), the HD is automatically turned off after a couple of minutes. That is bad. Think about you watching a movie. Then you feel like grabbing a coke and pausing the movie. When you come back, you resume the movie and the HD was stopped and has to start spinning again. I bet that the way they run hot and often start/stop can't do any good.

I think you should make backups of your important things to DVD's. New DVD's are supposed to be reliable (or more than they were a while back anyway). Just make sure to make two copies, with different brands for irreplaceable data (family pictures and movies). Don't bother with MP3 and movies, you can always download them again. It takes time, but it is, as I consider, replaceable.

By the way, what HD was it? So we all can avoid buying it in the future? :)

The HD was a Seagate, and normally I read only good reviews of those. Actually about that shutdown, most if not all external HD shut off (well don't shut off completely, but go into powersave mode) when not being used. Mine did that after 5 minutes of inactivity, but when I streamed movies off it, there was no interruption what so ever. As far as the movies, yea I could always recopy those, but even the MP3s I'm disappointed, because some of them were MP3s I downloaded years ago, that are next to impossible to find, (some Japanese Rock music, off of now defunct WinMX). I know that there are some MP3s that I will probably never see(listen to) again.

I think so, sort of, if you keep switching to the current type of disk. The good ones should have / should have had a good chance of lasting for as long as it took for DVDs to get as cheap as CDs or as long as it'll take for Blu ray or HDDVDs to get cheap. Of course, the manufacturer is not always the same as the brand.

I'm very wary of any hard drive without a fan, as well as frequent spin ups. Of course, that can only do so much.

One way to avoid this is to have a dedicated file computer with RAID1 (mirroring).

Well, if recordable media can't even last 5 years after it's been burned, I don't see whats the point in using them much. The theory is that they have a 100 year lifespan, some theory if they can't even last not even a quarter of that. For fscks sake, its not like I've been using zip discs, now those things are just terrible, completely unreliable. I know that there is flash now, but come on, lets be realistic, 250GB? Its more feasible to backup on one of the two things, either recordable media (DVD), or an external HD. Now whats the use if neither is reliable? There is no such thing as a reliable backup method then. :mad:

Well, there is magneto optical. But some CDRs can last more than 5 years. I'll check my really old ones.

Does anyone know if there's anything like nero scandisk for linux?

the light impressions, among other places, sells gold, guaranteed to last 50 years cd-rs.

And on the offchance it is ntfs, I have a really good rec program, and if it's fat, try getdataback for FAT.

That's is correct. 100% correct actually. Everything can fail and according to the Murphy laws, if it never fails, something must be wrong LOL :).

I'm pretty sure I have CD's much older than 5 years old which I kept in the basement and are still working though. Now, the amount of HD's that died on me are quite alarming. If one CD is not working, that means I have to replace 700MB of data. That is not that bad. If a HD fails, we are talking about 100, 200, 300 GB or more.

I mean, HD's have rotating parts. That is what sucks. Everything that runs/spin/turn or have magnetic crap will wear out faster. I'd really consider either DVD's or RAID for backup.

But as you said yourself, there really is no reliable way to backup stuff. And wait until one day, you lose your stuff and your backup won't work. That will sure make you mad :D. It is important to backup and test backups periodically to make sure they can be restored, when needed.

Tape archives, tape archives, tape archives, tape archives, tape archives, I can say it till we all turn blue. :)

They're supposed to be #1, and from what my friends who use them tell me, they really are that good.

But you should be fine with RAID1 and a weekly DVD-backup. Hard drives eventually fail, it's in their nature.

At like 10$ for a spindle of 25-50 (Depending on who made it), it's really not that expensive. If you have the cash, go for DVDRW, but even then, the RW capabilities eventually die...

But I'm still pretty sure CDs and DVDs can survive a long amount of time (~10 years) if properly stored (So in their own little cases in a box somewhere) and not used all the time.

I'm still using 10 year old CD-Rs and CD-RWs. They weren't even stored particularly carefully, just in their jewel cases. Memorex brand FWIW...

Some CDs of mine that I've tested:
11 year old pioneer disc - 0.72% unreadable
7 year old hitachi maxell disc - 0.54% damaged
7 year old TDK disc - perfectly readable
6 year old sony disc - 0.97% dameged and 0.36% unreadable
5 year old ritek disc - perfectly readable
and most surprising:
3 year old CMC magnetics disc - only 1.07% damaged

Memorex has been outsourcing for a while now, mostly to ritek I think.

Actually they're probably more like 12 or 13 years old, and still 100% readable. Not a single XIMAT or QIC tape of mine from that period still works BTW.

When it comes to long-term storage optical will always beat magnetic media.

Well, at this point, I am putting some of my faith back into recordable media. NOT MO, but as far as CD/DVDs are concerned, Memorex has been fairly decent, but Maxell seems to have been pretty good at data readability after a number of years. I think if I ever amass as much data as I did, next time I will probably get some company to make me some stamped discs, rather than burned, who knows. At this point I consider recordable media to be more reliable than HDs. I don't know about raid, and I don't see how tapes can be the most reliable since tapes wear out more, and I know I can't compare consumer tapes to pro backup tapes, but just like a vcr or cassette tape, it will degrade over time, so what effect would that have on data?

Jeebizz ... That really sucks. I also keep large archives of music and family pictures, as well as old code and other documents from as long as 15 years ago. I've lost hard drives, but I've been lucky enough to have made backups here and there, and thus far have been able to restore the data I most desire. (KNOCK ON WOOD!!!)

There are proponents of different types of backup. Tape backups are definitely the most well known for servers, and large amounts of data. CDs and DVDs are ok, but you can't plan on keeping them forever. (I've also got 10 year old CD's that I have no problem reteiving data from.) There's the RAID option. However, without a backup of your RAID, you still stand a chance of losing your data due to controller failure (not hard drive) or bad/evil software because it's on the same physical machine.

My current backup scheme is to keep live data on my server, and backing it up weekly to my wife's computer. This is 2 different computers, 2 different hard drives. That way if one of them is hit by lightening, or other hardware or software "anomaly", the data's still retrievable from the other system.

If you ever amass a similar amount of data than I would recommend two (or even three) external hdds. Keep one with your computer for constant access and rsync with the other one every week or so.

Your big mistake was only having one copy of your data. Just because you've labeled a drive as a backup doesn't magically make it invincible. Whatever media you use "redundancy" is the key word.

I just started to think about something... could it be that people complaining about unreadable CD/DVD's were using bad burning software back in the days? I mean, perhaps the programs did not burn properly back then or did not test the burned media, so we were "assuming" it was a good, 100% readable disc, when it was not. Not only because the media was bad (which should also be improved by now).

It is a known fact that some burners(software-wise) usually wrote crap to discs. Every application today is much better. Perhaps Nero is the only one that got worse and thinks that we all want to use Neroscout. I so hate Nero...

Sorry to hear of your loss Jeebizz. Sometimes the best lessons in life are the hard ones we face.

As mentioned already I think redundancy is the big thing. Keeping the data backed up in multiple locations, would be the best solution for ya. I most always recommend optical as a cheap backup solution just b/c you never know when a hard drive might go. And even going the extra mile with a RAID setup would be even better. This way the data you have backed up is in multiple locations...a few more hard disks and you actually have a portable copy. While I know everyone is different, I do think you should figure out what really does seriously need to be backed up and what doesn't. You have A LOT of info and IMO, not ALL of that needs to be on an external harddrive. Like some things could possibly be put on optical media and you can access those when you NEED them. And reburn the media eventually if you feel it may fail...I personally haven't had problems with aging optical media but then again my oldest disks are probably a year old. Every so often I just go through them and re-arrange how the data is layed out so I usually end up reburning em to new disks anyways. Disks are cheap so it's no biggie.

On a similar note, I have considered external hard disks but on another level I'd wonder if they'd fail quicker since they get shut off more. While this is kind of my theory does anyone out there have any insight into it?

the trend might be not to have any backup for most of the files/documents which are lets say more than a year old ... probably helps cutting down on possessivity which may "spin" the necessity of accumulating products/goods that we dont actually need ...


.

(Raid 1) <-- for hardware problems
+
DAT <-- for human errors and going backwards in time

Forget about the rest for long term backup
CDs/DVDs/External Disk have never been a reliable way of backing up. This is known in big companies.

Ah yes I forgot.. In case your house burns, put the DAT somewhere else :)

I suppose it's inevitable that each person gets caught at least once losing data without a usable backup. The more painful it is the less likely it will be to get caught again.

I use a system I call Paranoid Redundancy. I have a fileserver and two workstations. All user files reside on the server. Each workstation syncs once every hour with mirrordir to make a backup of the server data on the workstation. I do a full backup to DVD once a month, and incremental backups daily. The probability of losing all three hard drives at the same time is pretty small, so I am virtually 100% safe for having reliable data less than an hour out. The only times it will be necessary to restore from backup is if I should have a triple failure (like a house fire), or when I want something back that I have deleted or damaged. I keep copies of the backups at my daughter's house 15 miles away.

Don't ask what I experienced to become this paranoid.

Are you speaking in terms of quality, or just annoyed about neroscout? I have Nero Ultra 7 installed on my notebook, and I setup nero and bypassed neroscout all together. Perhaps you overlooked the option not to install it, but it is in the initial setup. I have never had any trouble with nero, and I like nero because of it's 'not holding your hand' gui approach, of course there is the wizard, but I never used it. I have seen the Roxio softwares, but thats just something I don't like, only because of the gui. I also remember one program, I don't even know if it is still around, but it was called NTI CDMaker. I used that when I first got my 2x cd burner back in the day.

Now as far as media is concerned, I think it also has something to do with the dye. I remember back in the days when CD-Rs first appeared, the dye color was kind of a dark greenish puke color, then it went to a somewhat blue color, until maybe late 1999 when it changed again to a lighter color (don't know how to describe it, just grab a CD-R nowadays and look.) I did have one disc, that took a lot of punishment, it was a blue dye CD-R I burned roughly 9 almost 10 years ago, still works, completely readable. It was a Verbatim disc. I guess though when it comes to CDs, perhaps for best results, choosing those 'cool color' discs, maybe is not a way to go and just choose the standard. I do have some black Memorex CD-Rs, but next time I will just be regular types. Who knows after another year or two they will change the dye on recordable DVD.

As far as burners, on my desktop, it is a TDK. That burner has been through quite a lot, and is still very reliable, I don't quite know the brand on my notebook. So I guess a quick recap, the best media to go with is the well known types, such as TDK, Memorex, Maxell, Verbatim, Sony, etc. Same with burners, though I can only think of TDK, Sony, and Pioneer off the top of my head right now.

I still find it hard to believe that tapes are more reliable than optical media, since tapes are more susceptible to data rot by magnetics, or just wear and tear. Of course as seen by everyone here though that optical media is still susceptible to data rot, I think it is at a much lower percentage rate, compared to tape, and even worse, data rot, or just like me, a complete failure of hardware.

Of course my mistake was putting all my data on that drive (putting my eggs in the proverbial basket), but the thing is when storage capacities get bigger and bigger, sifting through all that data gets more and more difficult. Not to mention keeping track of it.

Why use nero* for ripping when there is k3b? k3b is quite good, free and has technically overtaken nero since some time.

For optical, they may be safer but what's their price?

Yeah magnetical fields okay.. burry them then?

I don't think Nero is a burning program. It is more like a Cover Designer with a Indexed search engine and Avi2DVD converter. Besides it is also a Image editor program and a Media Player with support for Virtual Drive emulation. Maybe Nero 8 can also do me some coffee while I wait for my DVD to burn. Nero just happens to burn CD's/DVD's as well.

Regardless if I've missed or not neroscout during the installation (which I did twice), it should still provide a way to remove it afterwards. And it doesn't. I've tried to install the least features I could and it is taking over 190MB of disc space.

Personally, I think that "Nero" and "quality" should never be used in the same sentence. Unfortunately, Nero is still the best "Burning Software" available for Windows, together with CloneCD/DVD and Alcohol.

And the interesting thing is, half of the things Nero does, are not good enough. For example, Nero Vision is a joke. I still could not convert a .avi to DVD which worked. It either lags a lot when I put the converted disc on my DVD Player or the sound is not perfectly synchronized with the video. Meet Diko. I can convert avi to dvd and then burn the finished, converted files with Nero. No problems during playback.

Again, Nero Burning Rom is fine. Still, it is just a bunch of bloatware to me. K3B kicks the hell out of it, in my honest opinion. And it is free.

Ouch, Jeebizz. That sucks.

I'm getting more and more paranoid about my data these days. But at least my home folder can be completely tarred and compressed onto *one* DVD.
I wonder what I'll do when my home folder gets bigger?

I think it's worse when you have more documents, because you have more to wade through to decide what to backup. And it takes longer and longer.

Well I don't really have much of a problem with Nero. I never tried the Linux version, and although K3B is great, for some reason it doesn't burn discs at full speed on my notebook, even though DMA is set, however it burns full speed on my desktop. I dont pay much attention to the bloat of nero, though I agree it is getting rather bloated. I never use any of the other programs, just burning rom, and why the hell would I use their video converter, when tmpgenc is much better?

Also, I tend to burn a lot of dual layered discs, and I don't know how well K3B does dual layers. I think at this point, I am probably going to hope that blu ray overtakes hd-dvd because of capacity, though I am still not to happy about the many formats of DVDs. Unfortunately there is no real easy answer to backing up huge amounts of data (talking in terms to almost a quarter of a TB), and although there are also solid state hds out there, those are pricey, and don't hold probably as much as disc drives. Plus, solid states are essentially an EPROM, so now you have to worry about bit rot on those.

Good rule of thumb:
If it's spinning, it's not a backup.

I've got irreplaceable stuff (photos mainly) on DVD off-site (at work) and my /home, /etc and /var are backed up incrementally onto a USB drive using this method (modified to taste, obviously):
http://www.mikerubel.org/computers/r...s/#Incremental

Also, quick plug for my CD/DVD burning script should anyone want a CLI CD/DVD burner:
http://sourceforge.net/projects/multiburn/

Dave

nx5000, it's a good way to support a company trying to invest in Linux. I'd buy a copy of the latest Nero for Linux if I had the cash. Their products work, and if Nero succeeds, it shows other companies they have a chance too. First step is support on Free software, next is actually writing some ; )

Oh, and Nero's currently the only app that can burn HD-DVDs and possibly Blu-rays.

I have never seen any recordable hd-dvds, but I have come across BD-R and BD-REs, quite expensive. One disc, around $25 or so, for 30GB.

The name on the cover really doesn't mean that much. Instead you should worry about who it's actually made by. Where it's made tells you a lot about that. If it's made in Japan, it's probably made by Maxell or Taiyo Yuden. If it's made in Singapore, probably Mitsubishi Chemicals(also good). If it's made in Taiwan, it's likely Ritek or CMC magnetics. If it's made in India, it's by Moser Baer AFAIK.

Yes I agree. I didn't check the linux version nor its license but I guess (and understand due to their business model) that it's still proprietary. So... In a way I prefer if they stay out rather than come with proprietary software in the opensource world.
It's just two different worlds and I really don't install anything proprietary on my GNU/Linux, I would rather go back to windows.
The only optical disks I use are Cds :)

edit:
Maybe I haven't been in enough companies yet but I've never seen a professionnal backuping on optical disks. The tapes are quite efficient, cheap and quick but needs special readers for the home user..

Tape archives will lose their polarization, and thus their data over time. It happened in a lab I used to work in.

Also, anything full of moving parts (read: all hard drives) must fail eventually, regardless of brand etc. Some just go sooner than others...

Right. There's not really a foolproof backup method. Just the best method for your situation. Mechanical devices fail, optical devices degrade over time, and singularly don't hold a lot of data, magnetic media (tapes) also degrade, and can be very expensive when large capacity is a question.

At home my primary backup of archival data is to CDs and DVDs (non-rewritable), and backup of active data is to 2 synced data stores on 2 hard drives on 2 separate machines.

Spin off of original post, and rhetorical question: As far as my backed up archive data, I've got images and music. What is the guarantee that the JPG and MP3/OGG formats would be viable in 30-50 years? Each time a file is converted to a different format, some of its quality is lost. Of course, I probably won't care about my 80's hair band music when I'm 70 ... or will I? :cool:

Well, a slight good came out of this. A friend of mine had a majority of my music, ~70GB, plus I was lucky enough that he had copies of my vaction pictures, so I kinda got lucky. I am planning to burn all that data on DVDs as soon as I get them, but there is so much, I may have to invest in double layered discs. I have always had pretty good luck with the Verbatim brand, all the way back to CD-R, I never use RWs, a waste of time, and not reliable as WORMs. Even when I burn CD/DVDs, I always use only one session, and close it out/finalize it. I never do packet writing, just DAO all the way.

An external drive isn't an different than an internal so you shouldn't keep your only copy there, it should be a copy of the data on your internal drive.
Also you shouldn't rely on only one type of media, always use more than one type for backups, I use an external drive and cd-r/dvd-r's.
Third, never buy a "USB hdd" always buy the drive and usb enclosure seperate so you know what your getting. The pre-assembled ones always come with crap drives.

True that! Bad, Maxtor. Bad! Sorry to all of you with Maxtor drives, but ... blech!

My theory is to backup in triplicate. I send all of my pictures to at least two other family members on CD, I also keep a set for me and I also save data onto a large Hard Drive (Seagate) that I do not ever use except to make these one a month archives (Minimizes spin ups and downs), and because I am paranoid I also make a full annual backup including the OS and everything.

How far can we go to make sure that we have a trustworthy backup of the precious data of ours?
Well... let me put it this way:

"Working media" is the media storage where you process or in other ways use the data in daily basis.
"Backup media" is the media storage where you keep a copy of your data, safely away from any threats, ONLY accessed when updating or restoring the data.

Too many times I've stumbled over people who believes that storing their backups in a separate partition "D:\" is enough to keep them safe from harm, some even believes that a separate folder on the desktop is enough.

Pickup your backup media in your hand, take a good look at it and then answer those questions:

* Can it be physically disconnected/ejected? (a backup media shall ONLY be connected/inserted when accessed for update/restore)
* Can it be used with another computer? (a backup media shall be usable with the new computer in case that the old computer goes unusable)
* Can it be stored in a case/cover? (a backup media's lifetime is only as good as it's environment allows it to be)

If you answer "no" on anyone of those questions about your backup media, that means the backup media in mind isn't good enough for a proper backup.
External harddisks are useful if they fulfill the three conditions above.

Having the backup media (e.g. a hardisk) always connected and online (spinning) is one of the worse things you can do, since this WILL make it's lifetime significant shorter, even risking a mechanic failure.
All storage media WILL fail one way or another, it's just a matter of time.

List of potentional bad backup media:
* Memory cards (static electricity, even very small unnoticeable amounts of static electricity can render a memory card/chip totally useless. I can confirm)
* CD / DVD (re)writable (Very sensitive to their environment where heat, sunlight etc. and most of them are badly protected from scratches on the label-side. I can confirm)
* Floppy/Diskette (Is the anyone still using these? Anyhow, most of them got an extremly high risk for bad sectors. I can confirm)

I remember a scenario with floppies once upon a time:
I was about to restore a backup stored on a few (100) 3.5" floppies when I suddenly got an error message about some read error and then a message about aborting, retrying or failing the attempt, this at disk number 71. Tough luck!
Replacefloppy number 71 with the one from the second backup set (yes, I always had two sets of the same backup at that time).
Beginning to restore the data again from floppy 1 (the backup program couldn't continue after that error no matter what I tried.)
On the way to a restored system it shows up that disk nuber 95 also got some bad sectors ...

. . . *sigh* . . .

replacing ... starting over AGAIN.

Disk number 99 also broken, on both sets! *giving up after a few hours* . . . I litterally ... hate ... floppy disks!

SysGhost: 200 floppies?!? Good god... No wonder you hate floppy discs.

I backup everything onto WORM DVD. Approximately monthly, or whenever I'm doing something that might nuke my documents if I get it wrong.
I'm planning to get a fire resistant safe to store my backups in, but for now they're just stacked in a spindle case.

Currently, all my documents (including music) can fit onto a single layer DVD, so I think my self lucky. I don't have a lot of large documents, just lots of little files.

If you really want to backup to external hard drives, you need to have two external hard drives, and copy your data to both of them. Further, these are your backups, not your archive - these presume that you will also have your archive copies someplace else.

My commercial software resides on my workstation. A backup copy of my commercial software resides on an encrypted partition on my laptop (encrypted because the laptop travels with me and could get stolen). The other backup of my software resides on a pen drive which I usually keep in my car. After a major release, the release code goes on a dvd which goes to the bank safety deposit box, and every now and then I update that dvd.

The point? I *should* be covered no matter what happens; fire, theft, accidental data loss - none of those can separate me from at least one recent copy of my source code.

As for multimedia, pictures, etc., I presently back them up to two different external drives. Really what I do is scroll all that stuff off to one drive then synch that drive with the other one on a weekly basis. Not as proof against loss, but proof against the single point of failure loss that a failed HD represents.

What about rotating backups? One backup is never a smart choice when it comes to irreplaceable memories.

BTW, I had a hard drive fail that had our first family vacation in 15 years archived on it. And, about two years of photos, so I understand your pain.:cry:

[Edit]
I like the idea of backing up to several kinds of media: USB key, CD-R, DVD-RW, unplugged hard drive. I'd be wary of magnetic media though...one good swipe with the right magnet and your data is wiped clean.
[/Edit]