LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 02-18-2017, 01:32 AM   #1
idleman
LQ Newbie
 
Registered: Feb 2017
Posts: 2

Rep: Reputation: Disabled
ERROR "kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced"


Hi all,
I facing a trouble from the following errors:

[root@DATASRV(dat01-b)~]# grep error /var/log/messages
Feb 18 04:35:29 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337372
Feb 18 04:42:52 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337376
Feb 18 04:54:24 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337366
Feb 18 04:54:24 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337374
Feb 18 04:58:32 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337367
Feb 18 05:01:28 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337375
Feb 18 05:09:22 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337362
Feb 18 05:09:24 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337365
Feb 18 05:09:24 DATASRV kernel: EXT4-fs error (device dm-8): ext4_lookup: deleted inode referenced: 42337370
[root@DATASRV(dat01-b)~]#

These errors seems will only logged from mid-night to early morning, and very little in afternoon.

I found the privilege of some directories and files are ???:

[root@DATASRV(dat01-b)~]# debugfs -R 'ncheck 42337374' /dev/mapper/vg--dat009-lv--dat009 2>/dev/null
InodePathname
42337374 /acc-site1/420/100071420/0
^C
[root@DATASRV(dat01-b)~]# debugfs -R 'ncheck 42337368' /dev/mapper/vg--dat009-lv--dat009 2>/dev/null
InodePathname
42337368 /acc-site1/889/100035889/0
^C
[root@DATASRV(dat01-b)~]# ll /dat009/acc-site1/420/100071420/
ls: cannot access /dat009/acc-site1/420/100071420/0: Input/output error
total 0
d????????? ? ? ? ? ? 0
[root@DATASRV(dat01-b)~]# ll /dat009/acc-site1/889/100035889/
ls: cannot access /dat009/acc-site1/889/100035889/0: Input/output error
total 0
d????????? ? ? ? ? ? 0
[root@DATASRV(dat01-b)~]#

I using the RHEL 6.6 that installed in vSphere 5.5, Linux Kernel version is 2.6.32-431.
[root@DATASRV(dat01-b)~]# uname -a
Linux DATASRV 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

The partition /dat009 is mounted to the SAN switch and it is shared by a SAN.
[root@DATASRV(dat01-b)~]# df -khT /dat009
/dev/mapper/vg--dat009-lv--dat009 ext4 1.4T 21G 1.3T 2% /dat009
[root@DATASRV(dat01-b)~]#

I Google and found it should be a bug of RHEL and seems no patch to fix it. But I found a lot of suggestions are to run fsck to fix the issue.

However, our customer don't agree to umount the disk to run fsck, he think it is dangerous for the system, and he also don't want to occur service interruption.

Can you please help to share your experience and suggestions?

Thank you!
 
Old 02-18-2017, 04:29 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,713

Rep: Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549
The customer is always right.
However my attitude would be to point out they have accepted responsibility for using a filesystem known to have errors. So any future failure is not your fault, but theirs.

Some of my past managers have stated I have an attitude problem ... :shrug:
 
1 members found this post helpful.
Old 02-18-2017, 05:03 AM   #3
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 20 MATE
Posts: 8,048
Blog Entries: 5

Rep: Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917
A stitch in time saves nine.

You need to fsck that system. At the very least make sure that there are good recent backups in case things go even more awry.

svg0 is correct - explain the situation to the customer and hand responsibility back over if they refuse to play ball. You also have to determine why the customer thinks it would be "dangerous for the system" and provide arguments against that. If he/she is worried about the loss of income due to service interruption, point them towards the saying at the start of this post.
 
Old 02-18-2017, 09:22 AM   #4
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,547

Rep: Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082Reputation: 2082
The input/output errors suggest that a disk is failing. An unreadable inode will cause the "???????" result from ls. There should be messages logged in /var/log/messages at that time with more information about what happened.
 
Old 02-20-2017, 02:41 AM   #5
idleman
LQ Newbie
 
Registered: Feb 2017
Posts: 2

Original Poster
Rep: Reputation: Disabled
Thank you ALL,
We just run the fsck in the problematic partition and it is finished within 10 mins for 25 GB data.

However, the error have not be found again after fsck, but some files and directories are missing, only few files / directories can be moved to "lost+found". I restore the data from backup, it seems normal now.

Thanks!!
 
Old 02-20-2017, 05:48 AM   #6
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,713

Rep: Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549Reputation: 3549
This is what the customer was (probably) concerned about - fsck is to ensure the integrity of the filesystem not necessarily the files contained there-in. Good to see you have a solid backup regime.
 
Old 02-20-2017, 09:19 AM   #7
hydrurga
LQ Guru
 
Registered: Nov 2008
Location: Pictland
Distribution: Linux Mint 20 MATE
Posts: 8,048
Blog Entries: 5

Rep: Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917Reputation: 2917
Great news, Idleman.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
EXT4-fs error (device dm-0): ext4_lookup: deleted inode referenced: sandeep002gupta Linux - Software 9 10-11-2014 12:06 PM
bt5 boot fail :error ext2_lookup deleted inode referenced walvering Linux - Newbie 1 05-31-2012 01:26 AM
[SOLVED] "No root device found" error in FC14 after upgrade from ...97 to ...100 kernel PaulFC5 Fedora 5 11-04-2011 01:37 AM
kernel build error: deleted inode referenced sdfi Debian 11 10-23-2011 11:45 AM
Deleted inode referenced 83359... Koolassjoe Linux - Newbie 3 07-19-2011 06:14 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 02:24 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration