I'm running RedHat Enterprise Linux AS Release 3 (Taroon Update 6) on my
main server. Since August, the /home1 and /home2 partitions have been
using LVM on RAID5 with snapshot partitions being used for tape dumps.
The actual filesystems are ext3, which has journaling.

Twice now, I've gotten the following message on the console:

Message from syslogd@puma at Tue Nov 15 04:02:26 2005 ...
puma kernel: journal commit I/O error

Message from syslogd@puma at Tue Nov 15 04:02:26 2005 ...
puma kernel: Assertion failure in journal_flush_Rsmp_e2f189ce() at journal.c:1356: "!journal->j_checkpoint_transactions"

Message from syslogd@puma at Tue Nov 15 04:02:30 2005 ...
puma kernel: Kernel panic: Fatal exception

FYI, at 4:02am, the current snapshot is removed, a new one is created, and
a tape dump is begun. The effects of this error are that (1) I can no
longer "lvremove" the snapshot partitions, because the system says they
are in use, but they're really not; and, (2) I can't do tape dumps because
the "sync" system call hangs.

These can both be fixed by rebooting, but I don't want to be rebooting my
server in order to do tape dumps. I have two questions:

1. Should I be reporting this kernel error to anyone? If so, to whom and how?
2. Any suggestions about how to avoid this or fix it?

Thanks,

Scott

1. You should contact Red Hat.
2.?

I got the same error messages using Red Hat ES3 kernel 2.4.21-47.
When we used the kernel 2.4.21-32 this problem did not appear.


I had created a snapshot on the root file system.
And got the following errors while mounting the snapshot.


[root@nccdev01 bin]# mount /dev/Volume00/root_clone /root_clone
mount: block device /dev/Volume00/root_clone is write-protected, mounting read-only

Message from syslogd@nccdev01 at Tue Jan 9 16:07:00 2007 ...
nccdev01 kernel: journal commit I/O error

Message from syslogd@nccdev01 at Tue Jan 9 16:07:00 2007 ...
nccdev01 kernel: Assertion failure in journal_flush_Rsmp_e3ba0c6d() at journal.c:1356: "!journal->j_checkpoint_transactions"

Message from syslogd@nccdev01 at Tue Jan 9 16:07:00 2007 ...
nccdev01 kernel: Kernel BUG at journal:1356

Message from syslogd@nccdev01 at Tue Jan 9 16:07:00 2007 ...
nccdev01 kernel: invalid operand: 0000

Message from syslogd@nccdev01 at Tue Jan 9 16:07:00 2007 ...
nccdev01 kernel: Kernel panic: Fatal exception



The solution in my case to avoid the Kernel panic was to mount with "-t ext2"
[root@nccdev01 bin]# mount -t ext2 /dev/Volume00/root_clone /root_clone

I encountered this error as well, making a snapshot of /tmp. Any ideas on how I can get rid of the snapshot that was created when the panic occurred?

[root@garlic /]# lvremove /dev/vg00/SSlvol4
lvremove -- can't remove open snapshot logical volume "/dev/vg00/SSlvol4"

Kernel 2.4.21-47.EL, lvm-1.0.8-14 from the current CentOS 3.8, latest updates. Also tried removing the original volume (couldn't do it because a snapshot was open), vgchange -an (disallowed on active snapshot) and so on. Finally I made the snapshot fill up so it would become inactive, but that didn't work either. Here's what I'm left with:

[root@garlic /]# lvdisplay /dev/vg00/SSlvol4
--- Logical volume ---
LV Name /dev/vg00/SSlvol4
VG Name vg00
LV Write Access read only
LV snapshot status INACTIVE destination for /dev/vg00/lvol4
LV Status NOT available
LV # 10
# open 1
LV Size 2 GB
Current LE 64
Allocated LE 64
snapshot chunk size 64 KB
Allocated to snapshot 100.00% [511 MB/511 MB]
Allocated to COW-table 1 MB
Allocation next free
Read ahead sectors 1024
Block device 58:9

[root@garlic /]# lvdisplay /dev/vg00/lvol4
--- Logical volume ---
LV Name /dev/vg00/lvol4
VG Name vg00
LV Write Access read/write
LV snapshot status source of
/dev/vg00/SSlvol4 [INACTIVE]
LV Status available
LV # 4
# open 0
LV Size 2 GB
Current LE 64
Allocated LE 64
Allocation next free
Read ahead sectors 1024
Block device 58:3


Advice on how to whack these two LV's is greatly appreciated. Thank you.

Heart attack time when I saw this on a production server 30 minutes before a three day weekend....


I booted from the install CD. It found and mounted the mirror and the LVM.

HOME=/root chroot /mnt/target /bin/bash --login -i
lvremove /dev/vg00/swdev-bkups
(type yes)
exit
reboot


Fixed!

dgar - how exactly did you fix this?
I am having the same problem.

Hey,

This issue is fixed by Red Hat in U7. Check the below links:

http://rhn.redhat.com/errata/RHSA-2006-0144.html