For the first time yesterday I had these messages in my syslog

This went on for the next 12 hours reporting problems with various sectors.

Also see these


Finally ended with this set of messages

The disk then burst into life and started doing what it was supposed to do - rsync and encryption as the first stage of backing up to the cloud.

The disk is a WD 1TB Green SATA 6Gb/s 64MB 3.5" Hard Drive more than 5 years old. Though could be older.


I'm currently copying the contents to another external hard drive. No error messages being reported in syslog.

Once I've secured and validated the data and no errors have been reported should I do

1. e2fsck
2. and maybe smartctl

Would it be sensible to cut my losses NOW and replace the hard drive.

Alex

Based on the errors occurring at differing sectors and the age of the drive, I'd say it is due for replacement. If you post the output from "smartctl -A /dev/sdg" (wrapped in [CODE] ... [/CODE] tags, please, to preserve formatting), it will give a better picture of the drive's overall health.

At a minimum you will have to write zeros to the bad regions, which will cause the drive to reallocate the bad sectors to spares. If you continue using the drive, you will have to keep close watch on the bad sector counts. If bad sectors continue to develop, the drive will definitely need replacement.

Hard disks are commodity items - go get another one. You've done ok to get that long out of it. My data is more important than an old piece of hardware. You can play with working around shortcomings as suggested, but only if it doesn't compromise the data. I keep an old machine for this, never on my day-to-day system to avoid "finger-checks".

I've now managed to copy 164GB of data from the disk.

The cp command returned 115 error messages -

I take two backups -

1. rdiff-backup to this failed drive;
2. rsync to another drive,encrypting as I rsync. I then backup the encrypted data to the cloud. This disk appears to be error free, though it is a similar age to the failing drive.

As the failing disk is used solely for rdiff-backup, I'm not too concerned if I've lost rdiff-backup data.

I've tried smartctl and depending on how I address the failing drive I smartctl returns this
Don't know why if I address the drive as

• /dev/sdf SMART support is: Available
• /dev/bristol SMART support is Unavailable

I've convinced myself it's the same drive. Pulled the plug. The results are now

So I proceeded to test /dev/sdf

Then waited an retrieved the results



So bit confused as smartctl seems to imply that the disk is error free.

You give no clue as to what "/dev/bristol" might be. Is that some decryption mapping of the underlying /dev/sdf device? smartctl needs the raw device. It can't reach through the encryption layer to find it.

As for /dev/sdf: That shows 1455 bad sectors that will cause an I/O error when read. They don't show up in the test results simply because no long test has been run since the bad sectors developed. Running "smartctl -t long /dev/sdf" would almost certainly cause a failure to be logged. Rewriting those sectors would cause them to be reallocated to spare sectors, but a number that large is often a warning that the drive will continue to develop more bad sectors and could soon fail completely. Do not rely on the overall health statement from smartctl. That is generated by the firmware on the device, and bad sectors will not cause a health warning until the drive's supply of spare sectors is nearly exhausted. That is long past the point where the drive should have been replaced.

Note that an rdiff-backup archive is a fairly complex, and not terribly robust, database. Having various bit and pieces of it missing will cause history for those elements to be unavailable.

Not quite sure how to get the results so tried




After transferring the data to a new drive and after doing a "rdiff-backup --check-destination-dir" on the new drive did get back some reasonable results back. Anyway nothing can be trusted. So will look, with urgency, to carry on as is using a couple of USB drives plugged into my main machine then consider building a new machine to run a RAID backup solution. The Raid solution would be more a long term solution and might be too complex for my needs.

RAID is not a backup solution. It is for redundancy - say when a drive fails. If you issue "rm -rf" in a RAID environment your data is still gone.
You need a separate backup strategy. Always.

Those tests got interrupted right at the start. You have to let the test run to completion, without powering-off or rebooting.All "--check-destination-dir" does is check whether the most recent backup session failed or was interrupted before completion, and rolls back that session if that was the case.

One of the shortcomings of rdiff-backup is that it does not provide any good way to test the overall integrity of the archive. The only way to do that is to run with the "--verify-at-time" option for every session in the backup history. You can run several of those sessions in parallel in about the same time as a single session (there is a lot of commonality of disk access, so the kernel's buffer cache is a big win here), but it still takes a long time.

My current backup strategy is

1. to rsync to an external drive called "southsea" encrypting as I go then transfer the files from that external drive, using s3cmd, to the Amazon cloud. I don't backup all files only those that I would miss if something went horribly wrong
2. to rdiff-backup to an external drive called "bristol". This I use if I've changed a file and some days later want to backout the change to a specific date. Might be a few changes back.

It was "bristol" that failed the other day so no great loss. I will commission a new drive and start the rdiff-backup from day 1. I would lose the history of any changes I've made to files, usually scripts/programs in the past, at the moment that doesn't bother me.

If "southsea" had failed I will commission a new drive and start the rsync process to repopulate the drive. If I've got the S3cmd rules set correctly it would only transfer the changed files to Amazon as most of the files will have the same date/time stamp and md5sum.

If a hard drive on my production machine(s) failed I would get the files back from either "bristol", "southsea" or Amazon whatever was the most appropriate. i.e. hard drive failure or building burning down.

I've reread the Raid documentation and now consider it not to be more appropriate for my situation.

I'm going to put "bristol" and "southsea" out to pasture both are of a similar age, install new drives and see if the strategy I've described above works.