LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 01-14-2009, 03:56 AM   #1
farrukhndm
Member
 
Registered: Mar 2008
Location: Pakistan
Distribution: Fedora,Centos,RHEL
Posts: 75

Rep: Reputation: 15
Question Urgent Help Please......Fedora 8 Mail Server Crash !!


Dear All,
OS: Fedora Core 8
Yesterday till night everything was working fine,but in mornnig when i came office,and check Our servers was turn off due to some electric Issue.
When i trund on them i got these error and every thing was sturck,,
My Mail server ,proxy server is hosted on that..
Please guide me what i have to do..
here are some errors i got when shutting down my system

When i shutdown my linux by
#init 0

my Machine Struck and goet these errors

etc/rc.d/init.d/funciton : line 265 :2649 Segmentation fault rm-f " ${pid_file: -/var/run/$base.pid}"
etc/rco.d/K03yum -updated: line 41 :2650 segmentation fault rm-f /var/lock/subsystem/yum-updated
etc/rc.d/rc: line 53 :2651 Segmentation fault Lc_All=c egrep -q " *init/d/funcations"$i
stopping atd: stopping atd: [FAILED]
ETC/RC.D/RC:LINE 53:2658 sEGMENTATION FAULT
lC_ACL=C EGREP -Q " ** INIT.D FUCATION " SI
STOPPING CONSOLE kIT: STOPPIGN CONSOLE KIT : : [FAILED]
STOPPING USERMIN : STOPPOING USERMIN SERVER IN /USER/LIB/EXEC/USERMIN

and system is handup and dosn't shutdown..


My postfix,dovecot,squid are not working...


When resart these services i got errors


[root@mail ~]# /etc/init.d/dovecot restart
/etc/init.d/functions: line 256: 2060 Segmentation fault rm -f "${pid_file :-/var/run/$base.pid}"
/etc/init.d/dovecot: line 26: 2061 Segmentation fault rm -f /var/lock/subs ys/dovecot

Starting Dovecot Imap:


-------------------------

[root@mail ~]# /etc/init.d/dovecot restart
/etc/init.d/functions: line 256: 2060 Segmentation fault rm -f "${pid_file :-/var/run/$base.pid}"
/etc/init.d/dovecot: line 26: 2061 Segmentation fault rm -f /var/lock/subs ys/dovecot

Starting Dovecot Imap:



---------------------------

[root@mail ~]# /etc/init.d/dovecot restart
/etc/init.d/functions: line 256: 2060 Segmentation fault rm -f "${pid_file :-/var/run/$base.pid}"
/etc/init.d/dovecot: line 26: 2061 Segmentation fault rm -f /var/lock/subs ys/dovecot

Starting Dovecot Imap:


------------------------


[root@mail ~]# /etc/init.d/dovecot restart
/etc/init.d/functions: line 256: 2060 Segmentation fault rm -f "${pid_file :-/var/run/$base.pid}"
/etc/init.d/dovecot: line 26: 2061 Segmentation fault rm -f /var/lock/subs ys/dovecot

Starting Dovecot Imap:


-----------------------

[root@mail ~]# /etc/init.d/dovecot restart
/etc/init.d/functions: line 256: 2060 Segmentation fault rm -f "${pid_file :-/var/run/$base.pid}"
/etc/init.d/dovecot: line 26: 2061 Segmentation fault rm -f /var/lock/subs ys/dovecot

Starting Dovecot Imap:


========================
Is Recovery Possibele.................Help please..

i have backup of /etc/ & /home

waiting for your acknowledgment on Urgent bases as our Whole Organization Mailing system & Internet is stopped.
 
Old 01-14-2009, 04:04 AM   #2
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
Hi,

Have you had a look inside the init script /etc/init.d/dovecot?

It looks like it's trying to stop the 'running pid and remove the file but the file not being there causes a seg fault.

Go over the script, check what it is expecting and maybe touch the files it's looking for.

We'll need a bit more info to help you.

You might also force a filesystem check on reboot.
 
Old 01-14-2009, 04:21 AM   #3
farrukhndm
Member
 
Registered: Mar 2008
Location: Pakistan
Distribution: Fedora,Centos,RHEL
Posts: 75

Original Poster
Rep: Reputation: 15
how to Recover my system ?????

Quote:
Originally Posted by eco View Post
Hi,

Have you had a look inside the init script /etc/init.d/dovecot?

It looks like it's trying to stop the 'running pid and remove the file but the file not being there causes a seg fault.

Go over the script, check what it is expecting and maybe touch the files it's looking for.

We'll need a bit more info to help you.

You might also force a filesystem check on reboot.
Is there any recover process???????

how to check file system for As i have configured raid there,
/dev/md0,/dev/md1 & /dev/md2

Please check am i right

boot in init level 1 and then
fsck /dev/md0 ??????

or in init level one , i have to umount all partion ??????
 
Old 01-14-2009, 04:46 AM   #4
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
No, simply do
Code:
# shutdown -rF now
 
Old 01-14-2009, 04:53 AM   #5
farrukhndm
Member
 
Registered: Mar 2008
Location: Pakistan
Distribution: Fedora,Centos,RHEL
Posts: 75

Original Poster
Rep: Reputation: 15
how to use fsck on Raid volume ????

Quote:
Originally Posted by eco View Post
No, simply do
Code:
# shutdown -rF now
Same problem ..
Hangup in sotopping user min : stopping userminserver in /user/libexec/usermin


Please guide me how to run fsck on init level 1 or in rescue mode ??? help please....
 
Old 01-14-2009, 05:03 AM   #6
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
OK, the aim is to check your entire file system so if you are using grub.

Boot the box and in the grub menu, chose the single user option. If you don't have one. type e to edit and add the word 'single' at the end of the kernel line.

It will boot in single mode. Now you can start checking your filesystem by hand with a

Code:
# fsck /dev/mdX
but you might prefer to simply

Code:
# shutdown -rF
so that this type it checks the fs for you at boot.

Let me know how it goes, I'll try to keep an eye open for you. Back at 14h though.
 
Old 01-14-2009, 06:08 AM   #7
robertjinx
Member
 
Registered: Oct 2007
Location: Prague, CZ
Distribution: RedHat / CentOS / Ubuntu / SUSE / Debian
Posts: 749

Rep: Reputation: 73
You sure you weren't "hacked" ?! Stuff like this can happen when an attacker, gets in the system, gets root and installs a rootkit. The rootkit remove commands like "rm", "ls" and so on. There is a possibility too, you should try to check the system, with lets say chkrootkit or something.

Good luck
 
Old 01-14-2009, 06:29 AM   #8
ilikejam
Senior Member
 
Registered: Aug 2003
Location: Glasgow
Distribution: Fedora / Solaris
Posts: 3,109

Rep: Reputation: 97
Looks like your 'rm' binary is corrupted. Since Fedora 8 defaults to journalled ext3, it's unlikely that an fsck will sort this out (the filesystem shouldn't become corrupted by a power outage in the first place). Give fsck a go, but I forsee a reinstall in the very near future, since you've only got /etc and /home backed up.

Dave
 
Old 01-14-2009, 06:39 AM   #9
farrukhndm
Member
 
Registered: Mar 2008
Location: Pakistan
Distribution: Fedora,Centos,RHEL
Posts: 75

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by eco View Post
OK, the aim is to check your entire file system so if you are using grub.

Boot the box and in the grub menu, chose the single user option. If you don't have one. type e to edit and add the word 'single' at the end of the kernel line.

It will boot in single mode. Now you can start checking your filesystem by hand with a

Code:
# fsck /dev/mdX
but you might prefer to simply

Code:
# shutdown -rF
so that this type it checks the fs for you at boot.

Let me know how it goes, I'll try to keep an eye open for you. Back at 14h though.


-----------------
After above commands ...

and rebooting ans scanning my mdX

echo ${CONFIG##*/}
2400 Segmentation fault | grep -q '[^g]-'
./network-functions: line 66: 2415 Done echo ${CONFIG##*/}
2416 Segmentation fault | grep -q '[^g]-'
./network-functions: line 216: 2420 Done LC_ALL= LANG= ip -o link
2421 Segmentation fault | grep -q $1
./network-functions: line 216: 2424 Done LC_ALL= LANG= ip -o link
2425 Segmentation fault | grep -q $1
./network-functions: line 385: 2426 Done LC_ALL=C iwconfig $ 1 2>&1
2427 Segmentation fault | grep -q "no wireless extensions"
Error for wireless request "Set Encode" (8B2A) :
SET failed on device eth0 ; Operation not supported.
/etc/sysconfig/network-scripts/ifup-eth: line 297: 2441 Done LC_ALL=C ip addr ls ${REALDEVICE}
2442 Segmentation fault | LC_ALL=C grep -q "${IPADDR}/${PREFIX}"
./network-functions: line 66: 2470 Done echo ${CONFIG##*/}
2471 Segmentation fault | grep -q '[^g]-'
./network-functions: line 66: 2491 Done echo ${CONFIG##*/}
2492 Segmentation fault | grep -q '[^g]-'
[ OK ]
[root@mail ~]# /etc/init.d/httpd status
httpd dead but pid file exists
[root@mail ~]#
[root@mail ~]#

-----------------------
[root@mail ~]# service httpd restart
/etc/rc.d/init.d/functions: line 256: 2589 Segmentation fault rm -f "${pid_file:-/var/run/$base.pid}"

/etc/init.d/httpd: line 65: 2590 Segmentation fault rm -f ${lockfile} ${pidfile}
Starting httpd: [ OK ]
[root@mail ~]# service httpd restart
/etc/rc.d/init.d/functions: line 256: 2624 Segmentation fault rm -f "${pid_file:-/var/run/$base.pid}"

/etc/init.d/httpd: line 65: 2625 Segmentation fault rm -f ${lockfile} ${pidfile}
Starting httpd: [ OK ]
[root@mail ~]#

[root@mail ~]# service httpd restart
/etc/rc.d/init.d/functions: line 256: 2589 Segmentation fault rm -f "${pid_file:-/var/run/$base.pid}"

/etc/init.d/httpd: line 65: 2590 Segmentation fault rm -f ${lockfile} ${pidfile}
Starting httpd: [ OK ]
[root@mail ~]# service httpd restart
/etc/rc.d/init.d/functions: line 256: 2624 Segmentation fault rm -f "${pid_file:-/var/run/$base.pid}"

/etc/init.d/httpd: line 65: 2625 Segmentation fault rm -f ${lockfile} ${pidfile}
Starting httpd: [ OK ]
[root@mail ~]#


root@mail ~]# /etc/init.d/dovecot restart
/etc/init.d/functions: line 256: 2731 Segmentation fault rm -f "${pid_file:-/var/run/$base.pid}"
/etc/init.d/dovecot: line 26: 2732 Segmentation fault rm -f /var/lock/subsys/dovecot

Starting Dovecot Imap:
 
Old 01-14-2009, 06:49 AM   #10
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
Hi,

Looks like ilikejam might well be right.

Try to type the following to see if these commands exist

Code:
$ for i in rm echo ls grep; do ls -ld `which ${i}`; done
This should tell you if the commands still exist and when they where last modified (which might be important).

Apache is stopped because a pid file is still present but the service is stopped.
Quote:
# /etc/init.d/httpd status
httpd dead but pid file exists
Try removing the pid file manualy from /var/run/
Then start the httpd service and see if you still get an error.

If that works, you might go back to /var/run/ and see what other files are there that should not.

Keep us updated.
 
Old 01-14-2009, 08:08 AM   #11
farrukhndm
Member
 
Registered: Mar 2008
Location: Pakistan
Distribution: Fedora,Centos,RHEL
Posts: 75

Original Poster
Rep: Reputation: 15
decided to reinstall my server

Thanks... for helping....as all Services are giving errors "Segmentation fault"

So now i have decided to reinstall my server from start.

pleas tell guide me regarding to backup which i need to take before uninstalling my previous system

My installed Application was.
Mail server (Postfix,Openwebmail,Dovecot)
Proxy server (squid)

i have copied its /etc & /home whether i need /var ???????
and any other directory to restore it on new system.

Please guide me thanks..

If there is any other method to recover in this situation of Corruption in file system .

thanks.. and waiting..
 
Old 01-14-2009, 08:16 AM   #12
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
If I where you I'd backup all directories apart from /proc, /dev, /sys, ... This way you'll be sure not to miss anything. You should also make sure the backup is good before wiping your disks clan!

You might also need to dump your databases if any, ...

How was your mail setup configured? do you use databases, ldap, ...? Maybe take a list of all packages and their version.

Have you documented your installation? If not, it could take you a while to get it all back in place.

I hope you get back up and running soon.
 
Old 01-14-2009, 08:55 AM   #13
farrukhndm
Member
 
Registered: Mar 2008
Location: Pakistan
Distribution: Fedora,Centos,RHEL
Posts: 75

Original Poster
Rep: Reputation: 15
why i need to backup /proc /sys

yes.. i have everything documented may .still no database running there.
.hope i will do it.
Tonight i am not going back to home as i have to up my server.

Pleas tell me that why i need to backup /proc /sys ??? please guide me thanks & also tell me whether i need to backup /var ??? or just take backup of /www where my mail server contacts are saved

in Recovery process please tell that..

1)after installing Fedora 8 i replace my whole /etc/ (which is backup) earlier ??? and www backup earlier.. whether they will work

Now i want to virtulizaion .. you have experiance of it.i have check out XenServer 5.0 but it dosn't support 32 bit.
I am doin loading Vmware Server 2 (rpm) what you say about it??
or ESX Server it is not free


please guide me ....thanks..
 
Old 01-14-2009, 09:41 AM   #14
eco
Member
 
Registered: May 2006
Location: BE
Distribution: Debian/Gentoo
Posts: 412

Rep: Reputation: 48
Exclamation

Hi,

OK, let's start with virtualization. It's a good idea but it might not be the best time to start.

Xen takes a bit of work and you need to plan how you want to set up your virtual domains.

vmware Server is ok but I wouldn't use it for production, just testing. ESX is soooo expensive I wouldn't bother unless you have a lot of $ to spare.

What I meant to say is backup everything except /dev /sys /proc.

To restore, you can't just dump /etc over everything or you'll mess up your system and might even end up with the same errors. Instead use the backed up data to refer to your old config files, such as email server configuration, ... so that you can reinstall with the same specs as before.

Once all is setup, you need to find out how to recover your emails, ... It might be enough to put them back in the same place but you need to do some research. In fact, you might be better to do the research before you start working on the new server and lose the possibility to do some actions.

If for example, you backup your data, wipe the disks and then read you should have dumped a db so that you could recover from it, you'll have to do a lot of work to get back the data.

So, my advice is,
Forget about vm for now
Backup your data, maybe even make an image.
Check your backup(s) are good!
Reinstall the system
Configure the applications
migrate the data (emails, etc) to the new server.
When all is well and working again, backup your server as above. All but /proc,/dev,/sys.

Best of luck.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Please help! Urgent ! Mail server being used for sending spam kumar_79v Linux - Server 1 08-30-2008 12:22 AM
Urgent ! our mail server is unable to receive mail from Other domains! farrukhndm Linux - Server 6 07-02-2008 05:40 AM
URGENT!!! Setting up a mail server with Red Hat 9 Superman Linux - Newbie 8 05-16-2007 05:02 AM
Urgent help for Postfix mail server (please Read it) pushpraj Linux - Server 4 09-17-2006 09:51 PM
urgent: mail server down 2 days I_AM Linux - General 2 07-29-2005 09:59 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 05:03 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration