Why can't I fix ext3fs always getting mounted read-only?

chadwick · 09-11-2005, 01:07 AM

I have a problem with my linux partition (ext3fs) which does not seem
to be fixed by using fsck. I am running the stable version of Debian
on an Averatec 3360-EG1 laptop with kernel 2.6.8.2-386.

The problem is that the system keeps going into read-only mode. I
have run fsck from knoppix on my root partition (/dev/hda2).
I've tried the following combinations:
e2fsck -fcy /dev/hda2
e2fsck -fp /dev/hda2
e2fsck -fccy /dev/hda2
e2fsck -t ext3 /dev/hda2 -fp
All of these seem to fix the problem in that it is possible to boot
back into the system without any errors that I can see. The first
time I tried this, fsck 1.38 in knoppix reported that errors were found and
so it went ahead and fixed them. However, after rebooting and running for a while, the whole file system goes into read only mode.

I have also tried to just skip the knoppix routine and reboot right
back into the problem partition, hoping it will detect and fix the
problem. It does detect that there was a problem and it runs fsck
1.37 (21-Mar-2005) to fix it. Once again the problem appears to be
fixed until after half an hour the whole file system is mounted
read-only. The following is an example of an extract from dmesg when
I do this.

==========================================================

EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
journal_bmap: journal block not found at offset 3084 on hda2
JBD: bad block at offset 3084
JBD: Failed to read block at offset 3081
JBD: recovery failed
EXT3-fs: error loading journal.
EXT2-fs: hda2: couldn't mount because of unsupported optional features (4).
Unable to identify CD-ROM format.
cramfs: wrong magic
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs warning (device hda2): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
EXT3-fs warning (device hda2): ext3_clear_journal_err: Marking fs in need of filesystem check.
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.

==========================================================

ide1 at 0x170-0x177,0x376 on irq 15
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
journal_bmap: journal block not found at offset 3084 on hda2
JBD: bad block at offset 3084
journal_bmap: journal block not found at offset 3252 on hda2
JBD: bad block at offset 3252
JBD: recovery failed
EXT3-fs: error loading journal.
EXT2-fs: hda2: couldn't mount because of unsupported optional features (4).
Unable to identify CD-ROM format.
cramfs: wrong magic
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs warning (device hda2): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
EXT3-fs warning (device hda2): ext3_clear_journal_err: Marking fs in need of filesystem check.
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.

=========================================================

Looking in /var/log/messages I don't find anything that tells me what to do.
I can see the following: (copied by hand)

Sep 10 17:28:22 localhost kernel: RAMDISK: cramfs filesystem found at block 0
Sep 10 17:28:22 localhost kernel: RAMDISK: Loading 4188 blocks [1 disk] into ram disk... |^H/^H-^H\^H (plus much more gibberish composed of ^H)
Sep 10 17:28:22 localhost kernel: VFS: Mounted root (cramfs filesystem) readonly.

I'm not sure what all this means. I really want to understand what's going on here and I really don't want to reformat the drive. Does anyone have any clues? It would be greatly appreciated.
Thanks.

freakyg · 09-11-2005, 07:05 AM

post your /etc/fstab
something is wrong here.......the fstab will help us figure it out..........

chadwick · 09-11-2005, 10:51 AM

My /etc/fstab cut and pasted is everything inbetween but not including the lines of ===, including
an end of line after the last line. Very close to the time that the problem started occurring I did edit /etc/fstab. I added the last line for /dev/hda1 and also added spaces between columns so that it was more readable.
=============================================================
# /etc/fstab: static file system information.
#
#
#<file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
/dev/hda2 / ext3 defaults,errors=remount-ro 0 1
/dev/hda5 none swap sw 0 0
/dev/sda1 /mnt/sda1 auto sync,noauto,user,exec 0 0
/dev/hdc /mnt/cdrom iso9660 ro,user,noauto 0 0
/dev/hda1 /mnt/windoze ntfs ro,user,noauto 0 0
=============================================================

This problem doesn't seem to require the use of any particular program for it to happen. I can be just typing away at emacs and all of a sudden I'm not allowed to save my file any more, or all of a sudden firefox will crash. Apart from that, all I can think of that I've been using is the window manager fvwm. Right around the time that it first occurred, I was trying to listen to some radio stations, possibly with mplayer, which for all I know is a red herring.

Thanks.

freakyg · 09-11-2005, 11:48 AM

Code:

/dev/hda2 / ext3 defaults,errors=remount-ro 0 1

why is that there?? that is the problem!!

chadwick · 09-11-2005, 01:32 PM

That statement was put there automatically by the debian installation and it didn't create any problems for three months. Doesn't the fact that it's being remounted read only mean that there is a potentially harmful error that could cause damage if it were left mounted with write permissions? I'm sure it could be something really minor that doesn't matter anyway, but I don't have enough experience to know whether that's the case. But maybe I'll give it a shot. Okay, thanks again for your help.

chadwick · 09-11-2005, 03:45 PM

Okay, I went into knoppix, edited my /etc/fstab and did the routine "fsck -fy /dev/hda2".
Now my /etc/fstab looks like:

Code:

# /etc/fstab: static file system information.
#
#<file system>  <mount point>    <type>   <options>                  <dump>   <pass>
proc             /proc            proc    defaults                      0       0
/dev/hda2        /                ext3    defaults                      0       1
/dev/hda5        none             swap    sw                            0       0
/dev/sda1        /mnt/sda1        auto    sync,noauto,user,exec         0       0
/dev/hdc         /mnt/cdrom    iso9660    ro,user,noauto                0       0
/dev/hda1        /mnt/windoze     ntfs    ro,user,noauto                0       0

As certain as my word is my honour, it is STILL remounting /dev/hda2 as readonly! Perhaps "defaults" does that anyway?

Nylex · 09-11-2005, 03:48 PM

Quote:

Originally posted by chadwick
As certain as my word is my honour, it is STILL remounting /dev/hda2 as readonly! Perhaps "defaults" does that anyway?

Nope, defaults sets a partition read-write.

sundialsvcs · 09-11-2005, 05:24 PM

It is curious, to be sure, but you're also using fstab options that I personally am not familiar with. (Each FS driver can have its own.)

So, let's eliminate some possibilities here...

"Zeroth," let's avoid the usual knee-jerk reaction of ...

A virus! A virus!

Absent some clear indication of the presence of a rogue program, let us assume there is not.

First, look at all the files in /var/log to see if there are any recent clues, such as reporting filesystem errors.

Next, boot into a CD-ROM such as Knoppix and do an fsck on all devices. Do every non-destructive test you can think of.

chadwick · 09-11-2005, 06:29 PM

Sounds like some serious

on my part.
By the way /etc/mtab claims it is rw:

/etc/mtab:

Code:

/dev/hda2 / ext3 rw 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs rw 0 0
usbfs /proc/bus/usb usbfs rw 0 0

but the read-only errors like
man: can't create a temporary filename: Read-only filesystem
keep on coming.

I'll try what you suggest.

tuxdev · 09-11-2005, 06:33 PM

try reading /var/log/syslog.
It may have hard drive errors.

aysiu · 09-11-2005, 06:33 PM

You may also want to consult a breakdown of what everything means in /etc/fstab:

http://www.tuxfiles.org/linuxhelp/fstab.html

AmbroiseF · 09-13-2005, 03:20 AM

Hello everyone.
I'm having the same problem as described in the first post : my file system swaps to read-only mode.

Here is my config:
Debian testing/unstable
Kernel 2.6.12 i-686
PIV 3GHz - 512Mo RAM
Debian installed on a Maxtor 40Go drive, plugged on an Adaptec PCI card as a supplementary device.
2 partitions : / and /home (the buggy one)

Here is the problem:
Everything went fine for almost a year, but now, after a certain time (depending on how the drive is running), my /home partition is falling back to read-only mode.
Here is what /var/log/kern.log says about it :

Quote:

Sep 12 20:18:06 localhost kernel: journal_bmap: journal block not found at offset 6156 on hdg6
Sep 12 20:18:06 localhost kernel: Aborting journal on device hdg6.
Sep 12 20:18:06 localhost kernel: journal commit I/O error
Sep 12 20:18:06 localhost kernel: ext3_abort called.
Sep 12 20:18:06 localhost kernel: EXT3-fs error (device hdg6): ext3_journal_start_sb: Detected aborted journal
Sep 12 20:18:06 localhost kernel: Remounting filesystem read-only

Well, if I unmount /home and tries fsck, fsck finds errors, fix them and then, everything works fine again once /home is remounted.
The problem is that fsck doesn't find any bad block, nor is there any sign of anything wrong on the disk itself.

I was wondering if anyone had found a solution to this, because the last post dates from may 2005 ...

AmbroiseF · 09-13-2005, 03:52 AM

What else should I check?
I don't think formatting /home to be a solution ...

Here is what I have found in /var/syslog

Quote:

Sep 12 12:12:12 localhost kernel: hdg: 0 bytes in FIFO
Sep 12 12:12:12 localhost kernel: hdg: timeout waiting for DMA
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: error=0x40 { UncorrectableError }, LBAsect=32392452, sector=32392452
Sep 12 12:12:12 localhost kernel: ide: failed opcode was: unknown
Sep 12 12:12:12 localhost kernel: end_request: I/O error, dev hdg, sector 32392452
Sep 12 12:12:12 localhost kernel: Buffer I/O error on device hdg6, logical block 26772
Sep 12 12:12:12 localhost kernel: hdg: 0 bytes in FIFO
Sep 12 12:12:12 localhost kernel: hdg: timeout waiting for DMA
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: error=0x40 { UncorrectableError }, LBAsect=32392452, sector=32392452
Sep 12 12:12:12 localhost kernel: ide: failed opcode was: unknown
Sep 12 12:12:12 localhost kernel: end_request: I/O error, dev hdg, sector 32392452
Sep 12 12:12:12 localhost kernel: Buffer I/O error on device hdg6, logical block 26772
Sep 12 12:12:12 localhost kernel: hdg: 0 bytes in FIFO
Sep 12 12:12:12 localhost kernel: hdg: timeout waiting for DMA
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: error=0x40 { UncorrectableError }, LBAsect=32392453, sector=32392452
Sep 12 12:12:12 localhost kernel: ide: failed opcode was: unknown
Sep 12 12:12:12 localhost kernel: end_request: I/O error, dev hdg, sector 32392452
Sep 12 12:12:12 localhost kernel: Buffer I/O error on device hdg6, logical block 26772
Sep 12 12:12:12 localhost kernel: hdg: 0 bytes in FIFO
Sep 12 12:12:12 localhost kernel: hdg: timeout waiting for DMA
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: error=0x40 { UncorrectableError }, LBAsect=32392455, sector=32392454
Sep 12 12:12:12 localhost kernel: ide: failed opcode was: unknown
Sep 12 12:12:12 localhost kernel: end_request: I/O error, dev hdg, sector 32392454
Sep 12 12:12:12 localhost kernel: Buffer I/O error on device hdg6, logical block 26773
Sep 12 12:12:12 localhost kernel: hdg: 0 bytes in FIFO
Sep 12 12:12:12 localhost kernel: hdg: timeout waiting for DMA
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: error=0x40 { UncorrectableError }, LBAsect=32392454, sector=32392452
Sep 12 12:12:12 localhost kernel: ide: failed opcode was: unknown
Sep 12 12:12:12 localhost kernel: end_request: I/O error, dev hdg, sector 32392452
Sep 12 12:12:12 localhost kernel: Buffer I/O error on device hdg6, logical block 26772
Sep 12 12:12:12 localhost kernel: hdg: 0 bytes in FIFO
Sep 12 12:12:12 localhost kernel: hdg: timeout waiting for DMA
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Sep 12 12:12:12 localhost kernel: hdg: dma_intr: error=0x40 { UncorrectableError }, LBAsect=32392454, sector=32392454
Sep 12 12:12:12 localhost kernel: ide: failed opcode was: unknown
Sep 12 12:12:12 localhost kernel: end_request: I/O error, dev hdg, sector 32392454
Sep 12 12:12:12 localhost kernel: Buffer I/O error on device hdg6, logical block 26773
Sep 12 12:12:12 localhost kernel: kjournald starting. Commit interval 5 seconds
Sep 12 12:12:12 localhost kernel: EXT3 FS on hdg6, internal journal
Sep 12 12:12:12 localhost kernel: EXT3-fs: mounted filesystem with ordered data mode.

Maybe this helps, but I have to admit that I don't exactly understand what these errors imply : simple bad sectors on the drive? (why fsck won't mark them then? or hard drive dying? or wrong position of the moon this month?)

chadwick · 09-13-2005, 02:01 PM

No I have not yet found a solution to this problem. I haven't been able to devote much time to it over the past couple days, and I still have no more clues as to what the problem might be.

As my laptop uses a Hitachi IC25N060ATMR04-0 hard drive, I downloaded the Hitachi drive fitness test (dft32_v405_b00.iso) and checked all 60 GB with the most thorough test on that disk. So far no errors have been found (the laptop's only 4 months old anyway).

I have also scanned my windows partition using Norton antivirus with definitions updated Sep 11. I found three instances of a threat called dialer.CarpeDiem, but I'm assuming that has nothing at all to do with anything. I also did a scandisk from within windows, but I'm not sure if it actually scanned the whole drive since windows thinks the drive is 23 Gigs. Maybe I'll try downloading a DOS boot disk with scandisk on it and be sure it does the whole drive.

I also looked through my /var/log files. I'm unable to discern what most of it means, but there's something in particular that looks irregular. Under /var/log/syslog, /var/log/kern.log and /var/log/messages I also see several occurrences of the following:

Code:

Sep  7 10:56:32 localhost kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Sep  7 10:56:32 localhost kernel: hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=82263076, high=4, low=15154212, sector=82263076
Sep  7 10:56:32 localhost kernel: end_request: I/O error, dev hda, sector 82263076

Sep  7 10:56:36 localhost kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
Sep  7 10:56:36 localhost kernel: hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=82263084, high=4, low=15154220, sector=82263084
Sep  7 10:56:36 localhost kernel: end_request: I/O error, dev hda, sector 82263084

This series of error messages occurs only on that day, which is the day that the problem first arose.

I haven't figured out yet what other nondestructive tests can be done.

Bonzodog · 09-13-2005, 02:30 PM

Just my own two-penneth, and correct me if I'm wrong here, but I'd say that to both of you (Chadwick & Ambroise F), that your Hard disks have bad sectors on them containing vital system info. It sounds alot like both are dying slowly, and a re-install might temporarily solve the problem, but it might not. Get new drives, and re-install your distros, mount the old disks on the side, and recover what you can from them of any vital data.