Hi All,

I accidentally ran fsck -A on Redhat 6 machine and and now unable to run any command shows below error msgs. It would be great if any one would suggest how to resolve this issue and proceed further..

[root@server ~]# df -h
-bash: /bin/df: Input/output error
[root@server ~]#

@server var]# df -i
-bash: /bin/df: Input/output error
[root@server var]#
----
[root@server ~]# cd /
[root@server /]# ls
[root@server /]# cd var
[root@server var]# ls
account cache crash cvs db empty ftp games gdm lib local lock log lost+found mail named nis opt preserve run spool tmp www yp
[root@server var]#
I can able to enter all directories under / but unable to find through ls under /..

---------- Post added 07-28-15 at 04:22 AM ----------

[root@server ~]# fsck -A
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda2 is mounted.

WARNING!!! The filesystem is mounted. If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.

Do you really want to continue (y/n)? yes

/dev/sda2: recovering journal
fsck.ext4: Bad magic number in super-block while trying to re-open /dev/sda2
e2fsck: io manager magic bad!
[root@server ~]# mount
-bash: /bin/mount: Input/output error
[root@server ~]# cd
[root@server ~]#
[root@server ~]# mount
-bash: /bin/mount: Input/output error
[root@server ~]# fsck
fsck from util-linux-ng 2.17.2
fsck.ext4: Unable to resolve 'UUID=601e2d6f-7a01-4200-b9c9-e26684716173'
[root@server ~]# df -h
-bash: /bin/df: Input/output error
[root@server ~]#

I could see logs at this time as below ..

[root@server ~]# cat /var/log/messages
Jul 28 02:02:37 server kernel: EXT4-fs error (device sda2): __ext4_get_inode_loc: unable to read inode block - inode=841, block=4294967347
Jul 28 02:02:37 server kernel: EXT4-fs error (device sda2) in ext4_reserve_inode_write: IO failure
Jul 28 02:02:41 server kernel: EXT4-fs error (device sda2): file system corruption: inode #8 logical block 1652 mapped to 1082996 (size 1)
Jul 28 02:02:41 server kernel: EXT4-fs error (device sda2): file system corruption: inode #8 logical block 1653 mapped to 1082997 (size 1)
Jul 28 02:02:41 server kernel: EXT4-fs error (device sda2): file system corruption: inode #8 logical block 1654 mapped to 1082998 (size 1)
Jul 28 02:02:41 server kernel: EXT4-fs error (device sda2): file system corruption: inode #8 logical block 1655 mapped to 1082999 (size 1)
Jul 28 02:02:41 server kernel: EXT4-fs error (device sda2): file system corruption: inode #8 logical block 1656 mapped to 1083000 (size 1)
Jul 28 02:02:41 server kernel: EXT4-fs error (device sda2): file system corruption: inode #8 logical block 1657 mapped to 1083001 (size 1)
Jul 28 02:02:41 server kernel: EXT4-fs error (device sda2): file system corruption: inode #8 logical block 1658 mapped to 1083002 (size 1)
Jul 28 02:02:41 server kernel: EXT4-fs error (device sda2): file system corruption: inode #8 logical block 1659 mapped to 1083003 (size 1)
Jul 28 02:02:46 server kernel: attempt to access beyond end of device
Jul 28 02:02:46 server kernel: sda2: rw=16, want=34359738368, limit=20480000
Jul 28 02:02:46 server kernel: attempt to access beyond end of device
Jul 28 02:02:46 server kernel: sda2: rw=16, want=34359738376, limit=20480000
-------------
Jul 28 02:02:46 server kernel: EXT4-fs error (device sda2): __ext4_get_inode_loc: unable to read inode block - inode=811, block=4294967345
Jul 28 02:05:39 server kernel: EXT4-fs error (device sda2): __ext4_get_inode_loc: unable to read inode block - inode=144, block=4294967303


Jul 28 02:05:47 server kernel: EXT4-fs error (device sda2): __ext4_get_inode_loc: unable to read inode block - inode=130, block=4294967303
Jul 28 02:05:57 server kernel: EXT4-fs error (device sda2) in ext4_reserve_inode_write: IO failure
Jul 28 02:05:57 server kernel: EXT4-fs error (device sda2) in ext4_reserve_inode_write: IO failure
Jul 28 02:05:57 server kernel: EXT4-fs error (device sda2) in ext4_reserve_inode_write: IO failure
Jul 28 02:05:57 server kernel: EXT4-fs error (device sda2) in ext4_reserve_inode_write: IO failure
Jul 28 02:05:57 server kernel: EXT4-fs error (device sda2) in ext4_reserve_inode_write: IO failure
Jul 28 02:05:57 server kernel: EXT4-fs error (device sda2) in ext4_reserve_inode_write: IO failure
Jul 28 02:05:57 server kernel: EXT4-fs error (device sda2) in ext4_reserve_inode_write: IO failure

Jul 28 02:06:30 server kernel: EXT4-fs error (device sda2): __ext4_get_inode_loc: unable to read inode block - inode=798, block=4294967344
Jul 28 02:06:37 server nagios: SERVICE ALERT: client.linux.com;Root Partition;CRITICAL;SOFT;1;DISK CRITICAL - free space: / 0 MB (0 inode=-55460581683379696%):
Jul 28 02:06:56 server nagios: SERVICE ALERT: localhost;Root Partition;CRITICAL;SOFT;1;DISK CRITICAL - free space: / 0 MB (0% inode=-55460581683379696%):

does this means inodes are full ? how to troubleshoot further any answers pls..

Re-format and either restore from backup or reinstall. As was promised, your filesystem is now severely damaged. Options for recovery are quite limited. If you want to try, you should first save a copy of the entire /dev/sda2 partition to a separate device. Then, you can see whether "fsck.ext4 -f /dev/sda2" is able to recover anything. If there are important files that still aren't recovered, photorec is about the only option.

Thanks for update nicholas. I would be re-formatting the disk now as there is no option to further troubleshoot and fix the issue.