LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Server (http://www.linuxquestions.org/questions/linux-server-73/)
-   -   Directories / Files turned into pipes ! (http://www.linuxquestions.org/questions/linux-server-73/directories-files-turned-into-pipes-854896/)

slinx 01-07-2011 06:10 PM

Directories / Files turned into pipes !
 
Hello, I've had this problem before, and it conked a server tonight. We have Dell PowerEdge servers with PERC controllers and SAS drives. One of the drives will fail, and even after rebuilding the drive, after reboot, the system is corrupted. It fails with the dreaded
Code:

Kernel panic - not syncing: Attempted to kill init!
I usually have to boot the server to a rescue CD ( Scientific Linux 4.7), and copy corrupted files from a working system.

Today I had a server do this, and upon mounting the bad partition, I see that the /lib directory has been turned into a pipe!
Code:

[root@slinux dm-0]# ls -l
total 196
drwxr-xr-x    2 root root      4096 Dec  6 16:44 bin
drwxr-xr-x    2 root root      4096 Feb 24  2010 boot
drwxr-xr-x    4 root root      4096 Feb 24  2010 dev
drwxr-xr-x  102 root root      12288 Jan  7 16:42 etc
-rw-r--r--    1 root root        296 Feb 24  2010 event.log
-rw-r--r--    1 root root          0 Jan  7 23:15 forcefsck
-rw-r--r--    1 root root          0 Jan  7 16:41 halt
drwxr-xr-x    2 root root      4096 Feb 24  2010 home
p-----S---  13  170 root 3624927232 Feb 24  2010 lib
drwx------    2 root root      16384 Feb 24  2010 lost+found
drwxr-xr-x    2 root root      4096 Mar  9  2009 media
drwxr-xr-x    2 root root      4096 Jan 21  2009 misc
drwxr-xr-x    3 root root      4096 Feb 24  2010 mnt
dr-xr-xr-x    2 root root      4096 Feb 24  2010 net
drwxr-xr-x    2 root root      4096 Feb 24  2010 opt
drwxr-xr-x    2 root root      4096 Feb 24  2010 proc
drwxr-x---    8 root root      4096 Jan  7 16:10 root
drwxr-xr-x    2 root root      12288 Dec  6 16:44 sbin
drwxr-xr-x    2 root root      4096 Feb 24  2010 selinux
drwxr-xr-x    2 root root      4096 Mar  9  2009 srv
drwxr-xr-x    2 root root      4096 Feb 24  2010 sys
drwxr-xr-x    2 root root      4096 Feb 24  2010 tmp
drwxr-xr-x    2 root root      4096 Feb 24  2010 usr
drwxr-xr-x    2 root root      4096 Feb 24  2010 var

Right now, I'm copying /lib from a working server. Rebooting now...


What causes this? Is there an easier way to recover from this kind of error?

Thanks!

Edit: I successfully recovered the server, just had to drop to a shell and fsck 3 times, after copying /lib from the other server. If anyone knows an alternate way of recovering a directory from becoming a pipe, please let me know. It looks like all the files were moved into the "pipe" and maybe could have been repaired.

Should I expect better stability if I were to put my system files in ext3 file systems directly on disk partitions, rather than using LVM?


All times are GMT -5. The time now is 01:25 AM.