SlackwareThis Forum is for the discussion of Slackware Linux.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Slackware: 14.2
Kernel: 4.4.29
LVM2 based disk management.
I would like to ask you how should I start debugging errors related to the disk?
Below is an example of the problem. It happened after:
1) removepkg kernel-modules
2) installpkg kernel-modules
3) sync
4) echo 3 > /proc/sys/vm/drop_caches
But I had also similar situation when rsync-ing Slackware mirror.
Perhaps get a new disk and restore from backup? Is this a conventional, rotating disk or an SSD? It is awfully unusual for writes to fail on a rotating disk unless the drive has exhausted its supply of spare sectors. For an SSD, becoming read-only is a common way for the drive to protect your data when the drive starts to fail.
What is the output from "smartctl -A /dev/sda"? (Please wrap it in [CODE]...[/CODE] tags to preserve formatting.)
For an SSD, becoming read-only is a common way for the drive to protect your data when the drive starts to fail.
However, I believe most will brick themselves when the system is next rebooted (don't really understand why they don't just go permanently into a read-only state)... at least that is what the SSD challenge from TechReport seemed to indicate. Most drives failed into read-only mode, but only until a reboot, after which the drive effectively bricks itself and data becomes irretrievable.
So, before any reboots on a suspected failing SSD, it is best to ensure that any necessary data is backed up.
The machine is an online.net's server, running just a couple (maybe 3) of months.
The disk is (or at least should be) fairly new and not used much. It's an HDD.
It looks like the problem starts when there is more write I/O, e.g. rsync, but I was rsync-ing before without problems.
The machine has 16GB of RAM and it's almost always fully used by buff/cache, if that matters.
Maybe it's only my bad feeling, but the shell prompt feels kinda sluggish, e.g. "free -m" takes sometimes seconds to execute. It used to be fast in the past.
BTW, I had to run fsck to boot the system at all again; there were those strange (TM) messages about orphaned inodes, something about bitmap, etc. Should I consider the filesystem/OS pending re-installation, e.g. could there be some file's permissions broken or some config file being corrupted? Re-installing is not a big deal, just time consuming.
BTW2, if I recall correctly, SMART was disabled on the disk and I enabled it with some smartctl switch some days ago, if that matters. Unfortunately I don't remember exactly what it was and .bash_history does not have it.
The filesystem errors are probably a result of the disk starting to fail. The 196 events aren't good, the 197 events are bad. See wikipedia for a description.
Me, I'd get a new disk - doesn't matter how old the current one is.
I have to agree that the disk doesn't look good. If those bad sectors were in some laptop that was getting bounced around while it was running I wouldn't get too excited about them, but it's not something you want to see in a server, especially one that's been running only ~118 days. Disks are cheap. Get rid of it.
The drive encountered write errors on certain sectors of the disk. The sector numbers are given in the log output, eg 9096736, 4272896, 8955624, 8955368, etc.
First thing to do is make a backup. Do it now. The drive could fail catastrophically. However, sometimes the number of "bad sectors" will stabilise, and you might get useful life out of the disk in the future. It all depends on whether the values in the critical SMART parameters above keep increasing.
Usually a disk will reallocate (remap) bad sectors on a write, but for a drive with 4096 byte physical sectors (a "4K drive"), it can only do this if you re-write the entire 4k sector. Most operating systems use 512-byte logical sectors, so there are 8 logical sectors per 4K drive physical sector.
See whether you have a "4k" drive by running "smartctl -a /dev/sda", and looking for this:
to check these sectors. What I'd do is work out the ranges of bad sectors around each error. For a 4k drive, you'd expect a multiple of 8 logical sectors surrounding each identified bad sector.
If you're feeling lucky, you can then use:
Quote:
hdparm --write-sector X /dev/sda
...which will write zeros into the "bad" sector. If you zero the entire 4k range (ie 8 sectors), it should provoke the drive to remap the bad physical sector. A subsequent read of the range should then succeed. If so, congratulations, you've "fixed" the bad sector.
Repeat for all the identified bad sectors...
The problem now is you have a filesystem with some random "holes" in it. These could be within user files (in which case the file will now be corrupted), or filesystem meta-data, or just in unallocated space (if you're lucky!). It's possible to perform filesystem "forensics" to determine what was allocated to those sectors, but it's not trivial. For now, just force fsck the filesystem and allow it to repair any structural errors. Keep all the logs, as they may be useful for forensics in the future.
Keep an eye on the SMART stats to see if any of items #5 & #196-198 continue to increase. If they do, it's probably time to replace the drive.
FWIW, on our storage platform we normally replace drives if they exceed 600 bad sectors. However, the storage is mirrored (RAID1) so there are two copies of everything enabling lost sectors to be restored from the mirrored disk.
Regards,
Marc
Last edited by MarcT; 11-04-2016 at 01:01 PM.
Reason: Fix loads of typos
Thank you all for your replies.
I reported the problem to online.net support and after investigation they replaced the disk*.
But I don't mind if you keep posting interesting stuff here.
I did have a look at SMART, but to be honest, it is really hard to make sense of it if you don't have experience.
Even the replies to this thread aren't straightforward (i.e. replace immediately vs might be recoverable).
*) I can access my old server in rescue mode to recover data, and there is already second server machine waiting to switch to.
I did have a look at SMART, but to be honest, it is really hard to make sense of it if you don't have experience.
Even the replies to this thread aren't straightforward (i.e. replace immediately vs might be recoverable).
There's some information about SMART in this thread, it may be useful in your case.
Usually a disk will reallocate (remap) bad sectors on a write, but for a drive with 4096 byte physical sectors (a "4K drive"), it can only do this if you re-write the entire 4k sector. Most operating systems use 512-byte logical sectors, so there are 8 logical sectors per 4K drive physical sector.
Good point. I hadn't thought of that. From the looks of it, the affected sectors are holding file system metadata (inodes). These days, all but the tiniest filesystems allocate 4096-byte blocks for data, so the issue of trying to write to a partial physical sector doesn't arise there. I find it a bit surprising that the inodes aren't read and written with that same block size.
Come to think of it, how did that inode ever get into memory if the sector containing it is bad?
This is a good question, I just had a disk failure, 0 reallocated sectors, yet disk had plenty of bad sectors. Maybe SMART is not that smart after all.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
