Hi,
I am encountering a wierd problem on my Linux Server. There has been 2 unexpected shutdowns in the last 10 days and the reason is unknown. The strange thing is that I have some other Linux servers, but it is only 1 particular server which shuts down.

The /var/log/messages files show the following entries.

Mar 27 07:49:17 icsinoierd013 kernel: ide-cd: cmd 0x25 timed out
Mar 27 07:49:17 icsinoierd013 kernel: hda: irq timeout: status=0xd0 { Busy }
Mar 27 07:49:17 icsinoierd013 kernel: hda: irq timeout: error=0x00
Mar 27 07:49:17 icsinoierd013 kernel: hda: ATAPI reset complete
Mar 29 04:02:35 icsinoierd013 cups: cupsd shutdown succeeded
Mar 29 04:02:36 icsinoierd013 cups: cupsd startup succeeded


Mar 29 04:02:36 icsinoierd013 syslogd 1.4.1: restart.
Mar 29 10:26:50 icsinoierd013 shutdown: shutting down for system halt
Mar 29 10:26:51 icsinoierd013 init: Switching to runlevel: 0
Mar 29 10:26:52 icsinoierd013 gpm[3548]: *** info [mice.c(1766)]:
Mar 29 10:26:52 icsinoierd013 gpm[3548]: imps2: Auto-detected intellimouse PS/2
Mar 29 10:26:52 icsinoierd013 cups-config-daemon: cups-config-daemon -TERM succeeded
Mar 29 10:26:52 icsinoierd013 haldaemon: haldaemon -TERM succeeded
Mar 29 10:26:52 icsinoierd013 messagebus: messagebus -TERM succeeded
Mar 29 10:26:52 icsinoierd013 atd: atd shutdown succeeded
Mar 29 10:26:52 icsinoierd013 cups: cupsd shutdown succeeded
Mar 29 10:26:52 icsinoierd013 xfs[3581]: terminating
Mar 29 10:26:52 icsinoierd013 xfs: xfs shutdown succeeded
Mar 29 10:26:52 icsinoierd013 gpm: gpm shutdown succeeded
Mar 29 10:26:52 icsinoierd013 sshd: sshd -TERM succeeded
Mar 29 10:26:53 icsinoierd013 sendmail: sendmail shutdown succeeded
Mar 29 10:26:53 icsinoierd013 sendmail: sm-client shutdown succeeded
Mar 29 10:26:53 icsinoierd013 smartd: smartd shutdown failed
Mar 29 10:26:53 icsinoierd013 vsftpd: vsftpd shutdown succeeded
Mar 29 10:26:53 icsinoierd013 xinetd[3500]: Exiting...
Mar 29 10:26:53 icsinoierd013 xinetd: xinetd shutdown succeeded
Mar 29 10:26:53 icsinoierd013 acpid: acpid shutdown succeeded
Mar 29 10:26:53 icsinoierd013 crond: crond shutdown succeeded
Mar 29 10:26:53 icsinoierd013 rpc.statd[3310]: Caught signal 15, un-registering and exiting.
Mar 29 10:26:54 icsinoierd013 nfslock: rpc.statd shutdown succeeded
Mar 29 10:26:54 icsinoierd013 irqbalance: irqbalance shutdown succeeded
Mar 29 10:26:54 icsinoierd013 portmap: portmap shutdown succeeded
Mar 29 10:26:54 icsinoierd013 kernel: Kernel logging (proc) stopped.
Mar 29 10:26:54 icsinoierd013 kernel: Kernel log daemon terminating.
Mar 29 10:26:55 icsinoierd013 syslog: klogd shutdown succeeded
Mar 29 10:26:55 icsinoierd013 exiting on signal 15


Any clues as to why this might be happening?
Regards,
Ravi

Looks like someone is shutting the server down, either manually or via CRON. Check all your cron jobs, for all users. Also, since you have the time/date a system was shut down, type in "last", and see if there are any users that log in just before the shutdown. Look at their shell histories, and see if they're doing it.

Most of the time, hardware-related problems that cause an orderly shutdown (temperature, UPS sending a shutdown signal, etc.), will leave very noticeable events in your syslogs, telling you WHY the system is shutting down.

Thanks for the reply TBone.

I am new to Linux Administration.

Well I checked the existing cron jobs using:
crontabs -l

And it didnt return any rows.

When I checked the users using:
last

There was no suspicious activity or logons.

Finally I started checking system logs using the command:
system-logviewer

When I check the system log then I see a lot of segment faults of the following nature:

kernel: frmweb[11078]: segfault at 0000000000697465e rip 00000000000f7dd92c1 rsp 0000000000000000ffff6794 error 4

Could this be an issue?

Do it as root, as well as a normal user. If you type in crontab -l as you, you'll only see YOUR jobs. You can also try to peek into /var/spool/cron/tabs (that's what it is on OpenSuSE), and see what files are in there, Cat them out...those are the cron entries for the various users.

Also check /etc/cron*, for jobs in there.

Wouldn't be suspicious necessarily...but if the same user had logged in before both shutdown events, I'd give them a deeper look. Check the .bash_history files for those users.

Do you happen to have Oracle on that server??

There are 2 files in /var/spool/cron/ which are applprod and oraprod and but the content is empty in both the files.



As

[root@icsinoierp011 cron]# cd /etc
[root@icsinoierp011 etc]# ls cron*
cron.deny crontab

cron.d:
sysstat

cron.daily:
00-logwatch 00-makewhatis.cron 00webalizer 0anacron certwatch logrotate mcelog.cron prelink rpm slocate.cron tetex.cron tmpwatch

cron.hourly:

cron.monthly:
0anacron

cron.weekly:
00-makewhatis.cron 0anacron
[root@icsinoierp011 etc]# vi crontab
[root@icsinoierp011 etc]# cd cron.daily/
[root@icsinoierp011 cron.daily]# ls
00-logwatch 00-makewhatis.cron 00webalizer 0anacron certwatch logrotate mcelog.cron prelink rpm slocate.cron tetex.cron tmpwatch
[root@icsinoierp011 cron.daily]# vi 0anacron
[root@icsinoierp011 cron.daily]# grep shutdown *
[root@icsinoierp011 cron.daily]# cd ../cron.monthly
[root@icsinoierp011 cron.monthly]# grep shutdown *
[root@icsinoierp011 cron.monthly]# cd ../cron.weekly
[root@icsinoierp011 cron.weekly]# grep shutdown *
[root@icsinoierp011 cron.weekly]#

Hence I believe that none of the cron jobs are causing shutdown. Please correct me if I am wrong.



Yes I have an Oracle Server running.

You are correct..doesn't look like the shutdowns are caused by a cron job. Sometimes that's the first place to look...I know I've cron'ned a shutdown for maintenance before, and forgot to remove it later.

Based on that error you mentioned, I've seen that on an Oracle server before, and it has caused a shutdown to occur. If these are totally random (not like EXACTLY a week apart, or something like that), I'd run the logs past Oracle, to see what they can make of them. Also, check with your DBA's...see if they've upgraded/changed any Oracle stuff, and see when they normally do data-loads or run reports. That could coincide with the shutdowns....