LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Newbie (https://www.linuxquestions.org/questions/linux-newbie-8/)
-   -   why did server shutdown (https://www.linuxquestions.org/questions/linux-newbie-8/why-did-server-shutdown-852490/)

qwertyjjj 12-26-2010 03:46 PM
why did server shutdown
 
According to admins my server had shutdown so they rebooted it.
How can I check what happened?
I checked the var/log/messages file but it only seems to start from when they rebooted.

They said they rebooted and uptime says 6hrs but the things that confuses me is that when I looked at it, squid and openvpn were already running (1 and 2 processes respectively) and I don't think they are in the boot process anywhere.

phpdba 12-26-2010 03:48 PM
Normally, with logrotate, there should be a messages.1 file or an archive file with a .Z extention within the same directory. You can either tail it or cat through it by the date and time.

qwertyjjj 12-27-2010 02:57 AM
Quote:
Originally Posted by phpdba (Post 4203605)
Normally, with logrotate, there should be a messages.1 file or an archive file with a .Z extention within the same directory. You can either tail it or cat through it by the date and time.
messages.1 has a record at Dec 25 22:47:59
messages next record is
Dec 26 00:01:52 server88-208-237-198 syslogd 1.4.1: restart.

Seems to be some time in between with no indication of why...

qwertyjjj 12-28-2010 05:01 AM
Any ideas what else I could check?

phpdba 12-29-2010 08:27 AM
You could try looking at any of the other logs in that directory to see what was running and for how long. As an example, you can look at the httpd logs to see if they were running during that time or were they seized up as well. I am wondering if a process went out of control and locked up the server (similar to what I have seen on some Oracle boxes) to the point of preventing the logs from writing. What puzzles me is your statement in regards to openvpn and squid. They are not in your init.d scripts or have you done a chkconfig --list squid to see what runlevels they run? Is it possible some other process starts them as a dependent process so they won't show up in the scripts (assuming they aren't there). The other place I would check is the rc.d directories just to make sure. My thinking is that if we can find the logs to the last process running, it is probably the process that caused the problem. This is no guarantee of what it might be, but it should help.

qwertyjjj 12-29-2010 08:46 AM
Quote:
Originally Posted by phpdba (Post 4206569)
You could try looking at any of the other logs in that directory to see what was running and for how long. As an example, you can look at the httpd logs to see if they were running during that time or were they seized up as well. I am wondering if a process went out of control and locked up the server (similar to what I have seen on some Oracle boxes) to the point of preventing the logs from writing. What puzzles me is your statement in regards to openvpn and squid. They are not in your init.d scripts or have you done a chkconfig --list squid to see what runlevels they run? Is it possible some other process starts them as a dependent process so they won't show up in the scripts (assuming they aren't there). The other place I would check is the rc.d directories just to make sure. My thinking is that if we can find the logs to the last process running, it is probably the process that caused the problem. This is no guarantee of what it might be, but it should help.
[root ~]# chkconfig --list squid
squid 0 off 1 off 2 off 3 on 4 on 5 on 6 off
[root ~]# chkconfig --list openvpn
openvpn 0 off 1 off 2 off 3 on 4 on 5 on 6 off
[root ~]#

I found both squid and openvpn in the rc.d/init/d folder so I'm guessing that they must start up on server reboot.

Does the last line in the fstab seem ok? squid would not be able to start without that so I guess it must have.
Code:

LABEL=/                /                      ext3    defaults        1 1
LABEL=/var              /var                    ext3    defaults        1 2
LABEL=/boot            /boot                  ext3    defaults        1 2
tmpfs                  /dev/shm                tmpfs  defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                  /sys                    sysfs  defaults        0 0
proc                    /proc                  proc    defaults        0 0
LABEL=SWAP-sda6        swap                    swap    defaults        0 0
/dev/sda2 /var/spool/squid reiserfs defaults,notail,noatime 1 2
I don't have the webserver running on the server most of the time.
These are all the startup programs, I could check each individual log?

Code:
/etc/init.d
/etc/rc.d/init.d
/etc/rc.d/init.d/NetworkManager
/etc/rc.d/init.d/SystemEmail
/etc/rc.d/init.d/acpid
/etc/rc.d/init.d/anacron
/etc/rc.d/init.d/apmd
/etc/rc.d/init.d/atd
/etc/rc.d/init.d/auditd
/etc/rc.d/init.d/autofs
/etc/rc.d/init.d/avahi-daemon
/etc/rc.d/init.d/avahi-dnsconfd
/etc/rc.d/init.d/conman
/etc/rc.d/init.d/courier
/etc/rc.d/init.d/courier-authlib
/etc/rc.d/init.d/cpuspeed
/etc/rc.d/init.d/crond
/etc/rc.d/init.d/dc_client
/etc/rc.d/init.d/dc_server
/etc/rc.d/init.d/dhcdbd
/etc/rc.d/init.d/dnsmasq
/etc/rc.d/init.d/firstboot
/etc/rc.d/init.d/functions
/etc/rc.d/init.d/gpm
/etc/rc.d/init.d/haldaemon
/etc/rc.d/init.d/halt
/etc/rc.d/init.d/heartbeat
/etc/rc.d/init.d/httpd
/etc/rc.d/init.d/httpd-matrixsa
/etc/rc.d/init.d/ibmasm
/etc/rc.d/init.d/ip6tables
/etc/rc.d/init.d/iptables
/etc/rc.d/init.d/irda
/etc/rc.d/init.d/irqbalance
/etc/rc.d/init.d/killall
/etc/rc.d/init.d/krb524
/etc/rc.d/init.d/kudzu
/etc/rc.d/init.d/lm_sensors
/etc/rc.d/init.d/lvm2-monitor
/etc/rc.d/init.d/matrixsa
/etc/rc.d/init.d/mcstrans
/etc/rc.d/init.d/mdmonitor
/etc/rc.d/init.d/mdmpd
/etc/rc.d/init.d/messagebus
/etc/rc.d/init.d/microcode_ctl
/etc/rc.d/init.d/multipathd
/etc/rc.d/init.d/mysqld
/etc/rc.d/init.d/named
/etc/rc.d/init.d/netconsole
/etc/rc.d/init.d/netfs
/etc/rc.d/init.d/netplugd
/etc/rc.d/init.d/network
/etc/rc.d/init.d/nfs
/etc/rc.d/init.d/nfslock
/etc/rc.d/init.d/nscd
/etc/rc.d/init.d/ntpd
/etc/rc.d/init.d/oddjobd
/etc/rc.d/init.d/openvpn
/etc/rc.d/init.d/pcscd
/etc/rc.d/init.d/portmap
/etc/rc.d/init.d/postfix
/etc/rc.d/init.d/pptpd
/etc/rc.d/init.d/psacct
/etc/rc.d/init.d/pure-ftpd
/etc/rc.d/init.d/rawdevices
/etc/rc.d/init.d/rdisc
/etc/rc.d/init.d/readahead_early
/etc/rc.d/init.d/readahead_later
/etc/rc.d/init.d/restorecond
/etc/rc.d/init.d/rpcgssd
/etc/rc.d/init.d/rpcidmapd
/etc/rc.d/init.d/rpcsvcgssd
/etc/rc.d/init.d/saslauthd
/etc/rc.d/init.d/single
/etc/rc.d/init.d/smartd
/etc/rc.d/init.d/squid
/etc/rc.d/init.d/sshd
/etc/rc.d/init.d/syslog
/etc/rc.d/init.d/sysstat
/etc/rc.d/init.d/tux
/etc/rc.d/init.d/webmin
/etc/rc.d/init.d/wpa_supplicant
/etc/rc.d/init.d/xfs
/etc/rc.d/init.d/xinetd
/etc/rc.d/init.d/ypbind
/etc/rc.d/init.d/yum-updatesd

phpdba 12-29-2010 11:44 AM
To turn off the squid and openvpn you simply need to run "chkconfig squid off" and do the same for openvpn. You can then check them by running the chkconfig --list command. As for the logs, you can check each one, but it may be simpler to grep through them for time stamps between the start and stop times you noticed in messages.1 file. There should be a good sed command for this as well, but I am pretty week on the sed/awk side at the moment. Someone else in the forum may be able to give you a quick sed command. I am taking a guess here, but you are running a version of SuSE? You may have posted it and I simply missed it so please forgive me if I did. There is also a possibility you may not find out for certain. The real question will be if this happens again. It gives you the ability to find a pattern (or in this case - a sort of pattern). Let me know how it turns out.


All times are GMT -5. The time now is 03:39 PM.