Server crashing once a day. CentOS 4.4

Kandahar · 09-25-2008, 04:57 AM

Hello,
I am having problems on a server which is constantly crashing once a day. (Some times it lasts two but rarely)

Its been going on for 2 weeks now and its the first problems I've had with this box for 2 years.
I run a network of Windows boxes because of the software we use here, so when it comes to Linux I completely fall over and don't know what to do to help track down this issue.

I am running top currently but nothing looks out of the ordinary.
Is there a way to monitor so I know when something spikes?
The load is ALWAYS below 1. So if anything goes higher then it should be that process.

When it crashes... It basically stays connected (I have a KVM/IP connected so I could post screenshots next time it happens - Probably tomorrow) - However you can't connect via SSH and all websites stop responding on it.
I think it responds to ping.

Any help will be very helpful. This is a web box so having it go down once a day is terribly dammaging.

Thanks

dyasny · 09-25-2008, 05:45 AM

/var/log/messages should have something

do you see any kernel panics or ooopses when the server crashes?

Kandahar · 09-25-2008, 06:32 AM

Here are things which jump out at me as being suspicious.

Code:

Sep 21 09:42:57 www sshd(pam_unix)[7295]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=196-209-58-100-rrba-esr-3.dynamic.isadsl.co.za 
Sep 21 09:42:58 www sshd(pam_unix)[7336]: check pass; user unknown
Sep 21 09:42:58 www sshd(pam_unix)[7336]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=196-209-58-100-rrba-esr-3.dynamic.isadsl.co.za 
Sep 21 09:43:01 www sshd(pam_unix)[7352]: check pass; user unknown
Sep 21 09:43:01 www sshd(pam_unix)[7352]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=196-209-58-100-rrba-esr-3.dynamic.isadsl.co.za 
Sep 21 09:43:02 www sshd(pam_unix)[7366]: check pass; user unknown

Lots and lots of them..

And then this happens:

Code:

Sep 21 09:43:39 www psa: Stopping Plesk:  succeeded
Sep 21 09:43:39 www named:  succeeded
Sep 21 10:43:39 www named[3214]: shutting down: flushing changes
Sep 21 10:43:39 www named[3214]: stopping command channel on 127.0.0.1#953
Sep 21 10:43:40 www named[3214]: no longer listening on 127.0.0.1#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: no longer listening on 77.74.1XX.XXX#53
Sep 21 10:43:40 www named[3214]: exiting

More things like this:

Code:

Sep 22 12:57:18 www sshd(pam_unix)[718]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=s030.silver.fastwebserver.de 
Sep 22 12:57:18 www sshd(pam_unix)[715]: check pass; user unknown
Sep 22 12:57:18 www sshd(pam_unix)[715]: authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=s030.silver.fastwebserver.de 
Sep 22 12:57:19 www sshd(pam_unix)[723]: check pass; user unknown

I have the messages file hosted if you would like to look at it.
I can PM you the link.

Kandahar · 09-25-2008, 06:47 AM

This also:

Quote:

"Sep 25 08:39:54 www kernel: Out of Memory: Killed process 32254 (httpd)."

salasi · 09-25-2008, 07:06 AM

Once you've got to this point:

Quote:

Sep 21 10:43:39 www named[3214]: stopping command channel on 127.0.0.1#953

(i.e., you are shutting down loopback#953), you can expect things to go wrong. And they do...

It looks as if this stops named working (surprise, surprise) and things get worse from there on in. I'm guessing the 77.74.1XX.XXX#53 is a legitimate name server for you to be trying to use for external name resolution, but when you do try it you get confused about what to do with the response, because the data normally flows along loopback and that isn't now working.

I'm sure I have known what Plesk is (but my feeble memory needs an extra stick of ram), but is it possible that:

Quote:

Sep 21 09:43:39 www psa: Stopping Plesk: succeeded

is the root cause?

Eventually

Quote:

"Sep 25 08:39:54 www kernel: Out of Memory: Killed process 32254 (httpd)."

it looks as if all memory becomes exhausted and the box has no option but to start shutting things down.

chrism01 · 09-25-2008, 09:16 PM

Re those attempted ssh logins:
uid=0 euid=0
means an attempt as root user. I hope you've disabled root login via ssh (see sshd config). There are thousands of bots on the net trying to break in that way. Always login as normal user and su or sudo up as needed.