How do you delete a corrupted file or directory?
 

First off, I'd like to say that Promise RAID controllers are WORTHLESS!!! Absolutely, ridiculously worthless.

To make a long story short, my raid controller has corrupted a bunch of data on my file server. Luckily, my nightly backup got most of it before it wnt down the tubes, but now, while I'm trying to restore, I can't overwrite any of the corrupted files, I can't even delete the files...

[root@uranium /job]# rm -rf HDKProjects
rm: cannot remove `HDKProjects': Permission denied

# ll
ls: HDKProjects: Permission denied
total 10

# rmdir HDKProjects
rmdir: `HDKProjects': Permission denied

... so, if root cannot delete the files, how can I get rid of them?

What filesystem are you using? I've seen tutorials out there for restoring corrupted files in a journaling filesystem like reiserfs.

Also if you are using ext2 or 3 there is the immutable flag, which could have been set when things were corrupted.
(edit)
Actually, not sure why that would give access denied for an ls... so probably not.

it's formatted reiser, but it's also RAID 5... don't know if that matters. Promise tech support said that when it's in a "critical state" (i.e. when only three drives are online), all files written will be corrupted when comes back to its normal state. Maybe it remaps the parity? I'm not sure.

I'll look into resier recovery tutorials. If anyone knows of one, please chime in.

Good god, you use a RAID 5 array? What the hell do you do with your system, control NORAD? <g>

How many HDDs do you have?

If one drive failed, your RAID controller should have transparently fixed it. If it didn't, get a new RAID controller. Promise should send you a new one. (Gateway would, I don't know other companies' policies)

If two failed, you are somewhat screwed and Promise is not to blame. 2 drives failing is beyond the capability of RAID 5 to restore. Or pretty much any RAID, now that I think about it. You can do two things:

1: Reformat all HDDs and restore from your backups. Easiest.
2: If you REALLY want to save the rest of what you did since backup: Reformat then restore the failed HDDs one at a time. This *should* save everything except what was on the failed blocks. Since these blocks have essentially been zeroed out on your failed drives, you should be able to r/w/delete them. They should no longer be so nastily corrupted, though they will be unusable. Restore them from backup.


2.5: ...I thought of a much better solution. Woot. Dump all the readable files to a copy of your old backup, rewriting wherever necessary. Then the noncorrupted files will be saved and you get the old versions of the corrupted files. FFR.

Whether 2 is possible depends on your HDDs, your RAID controller, and several acts of god. 2.5 is better. 1 is easiest.

hehe.. well, RAID 5 gices you the mos storage space while still providing some form of backup.. 4 250GB drives yields 750GB of usable space with 5, only 500GB of usable space with 0+1.
4 now. For some reason, only three came back online that last time it booted - probably an issue with the crappy controller card.
well, that's what I would have thought also, but that, apparently, is not the case. RAID 5 protects you when one disk fails, but not necessarily if one disk does not come online at boot-up. I feel that both cases are the same, however, Promise says that when one disk does not come online at boot, it's considered to be in a "critical state" and all data written while in a critical state is corrupted (This is what tech support told me over the phone). Personally, I think that is a load of crap.

One more reason to never buy a Promise controller again.

Thanks for the suggestions. I'm in the middle of backing up everything I can.. Being that it was formatted with ReiserFS, the ability exists to rebuild the file system tree which may recover my lost data. I'll give that a go after I backup everything that doesn't get backed up with my nightly minimal backup.

If a disk suddenly refuses to come online at boot, it's probably dead. It sounds like what happened is you had a total hardware failure of one and only one disk. What Promise told you is correct: the RAID array will fix a corrupted disk (ie if you took a magnet to one disk), but not a dead disk (ie if the head crashed). Also, any data you wrote after one HDD crashed will be unusable, since 1/4 of it wasn't written anywhere. It's possible that if you replace that disk altogether with a fresh disk of the exact same make/model/capacity, the other four disks will transparently fix the whole thing as if nothing ever happened --probably including everything you wrote after the crash! I'd try booting to a restore CD of some type (Knoppix?) and running a hardware diagnostic on each disk individually. This might require disassembling your RAID array and plugging in each disk one at a time as a slave so that you can view them as individual drives. Ugh. Unless your controller has some sort of debug mode or offers a diagnostic.

well, I might agree with you other than the fact that when I rebooted the machine, all four disks came back online. There was no catastrophic disk failure... the card did not see one disk for some reason, and when the disk did come back online, it did nothing to fix the old problem.

Have you looked at the files to see if the damage may have altered there attribute settings? Attribute settings can be such that even if permissions are set correctly no one, not even root can delete them.

To take a look at the attributes for files in a given folder use...

lsattr

If you determine that you need to alter the attributes, you can use the 'chattr' command to accomplish that task.

Good Luck