Hi all,
I facing a trouble from the following errors:

[root@DATASRV(dat01-b)~]# grep error /var/log/messages
Feb 18 04:35:29 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337372
Feb 18 04:42:52 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337376
Feb 18 04:54:24 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337366
Feb 18 04:54:24 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337374
Feb 18 04:58:32 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337367
Feb 18 05:01:28 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337375
Feb 18 05:09:22 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337362
Feb 18 05:09:24 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337365
Feb 18 05:09:24 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337370
[root@DATASRV(dat01-b)~]#

These errors seems will only logged from mid-night to early morning, and very little in afternoon.

I found the privilege of some directories and files are ???:

[root@DATASRV(dat01-b)~]# debugfs -R 'ncheck 42337374' /dev/mapper/vg--dat009-lv--dat009 2>/dev/null
InodePathname
42337374 /acc-site1/420/100071420/0
^C
[root@DATASRV(dat01-b)~]# debugfs -R 'ncheck 42337368' /dev/mapper/vg--dat009-lv--dat009 2>/dev/null
InodePathname
42337368 /acc-site1/889/100035889/0
^C
[root@DATASRV(dat01-b)~]# ll /dat009/acc-site1/420/100071420/
ls: cannot access /dat009/acc-site1/420/100071420/0: Input/output error
total 0
d????????? ? ? ? ? ? 0
[root@DATASRV(dat01-b)~]# ll /dat009/acc-site1/889/100035889/
ls: cannot access /dat009/acc-site1/889/100035889/0: Input/output error
total 0
d????????? ? ? ? ? ? 0
[root@DATASRV(dat01-b)~]#

I using the RHEL 6.6 that installed in vSphere 5.5, Linux Kernel version is 2.6.32-431.
[root@DATASRV(dat01-b)~]# uname -a
Linux DATASRV 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

The partition /dat009 is mounted to the SAN switch and it is shared by a SAN.
[root@DATASRV(dat01-b)~]# df -khT /dat009
/dev/mapper/vg--dat009-lv--dat009 ext4 1.4T 21G 1.3T 2% /dat009
[root@DATASRV(dat01-b)~]#

I Google and found it should be a bug of RHEL and seems no patch to fix it. But I found a lot of suggestions are to run fsck to fix the issue.

However, our customer don't agree to umount the disk to run fsck, he think it is dangerous for the system, and he also don't want to occur service interruption.

Can you please help to share your experience and suggestions?

Thank you!

The customer is always right.
However my attitude would be to point out they have accepted responsibility for using a filesystem known to have errors. So any future failure is not your fault, but theirs.

Some of my past managers have stated I have an attitude problem ... :shrug:

1 members found this post helpful.

A stitch in time saves nine.

You need to fsck that system. At the very least make sure that there are good recent backups in case things go even more awry.

svg0 is correct - explain the situation to the customer and hand responsibility back over if they refuse to play ball. You also have to determine why the customer thinks it would be "dangerous for the system" and provide arguments against that. If he/she is worried about the loss of income due to service interruption, point them towards the saying at the start of this post.

The input/output errors suggest that a disk is failing. An unreadable inode will cause the "???????" result from ls. There should be messages logged in /var/log/messages at that time with more information about what happened.

Thank you ALL,
We just run the fsck in the problematic partition and it is finished within 10 mins for 25 GB data.

However, the error have not be found again after fsck, but some files and directories are missing, only few files / directories can be moved to "lost+found". I restore the data from backup, it seems normal now.

Thanks!!

This is what the customer was (probably) concerned about - fsck is to ensure the integrity of the filesystem not necessarily the files contained there-in. Good to see you have a solid backup regime.

Great news, Idleman.