Nagios Notification for Reboots

ZiPPygEEk · 06-05-2009, 05:43 PM

I'm going to open a fresh thread on this issue, in hope that others have the same issue/question.

I'm looking to setup an alert for when a server reboots. I've posted on another thread regarding a similar issue, but I wanted to explain it.

I have PING setup right now, which does indeed work but the problem is it doesn't notify me for reboots. By the time the server comes back up on its own, no notification is sent out.

We have a server that occasionally will just reboot itself, so I wanted to know when this happens.

What do you guys recommend? A service to be watched? The check-host-alive feature? Which by the way I have setup, but it also won't notify me fast enough.

Any thoughts or direction???

Thanks,

ZiPPy

twantrd · 06-05-2009, 07:07 PM

You can lower the retry check interval and the max check attempts. However, if the server comes back online in less than a minute, I would suggest just writing a daemon that watches the log file in the real time and send you guys an email when it sees a reboot entry.

-twantrd

Matir · 06-05-2009, 07:15 PM

Have you considered writing a Nagios plugin that checks to see if the current uptime is less than the service check interval, and if so, sends an alert?

Samotnik · 06-05-2009, 07:19 PM

I think the best solution will be a simple messaging script in startup scripts of your machine. It will send you a message every time your server brings up after rebooting.

twantrd · 06-05-2009, 07:22 PM

Quote:

I think the best solution will be a simple messaging script in startup scripts of your machine. It will send you a message every time your server brings up after rebooting.

Actually, that's a better idea

Matir · 06-05-2009, 07:52 PM

Quote:

Originally Posted by Samotnik

I think the best solution will be a simple messaging script in startup scripts of your machine. It will send you a message every time your server brings up after rebooting.

Good call. KISS, right?

ZiPPygEEk · 06-08-2009, 05:04 PM

I have no idea how to script any of these suggestions. I tried setting the retry interval, but that still doesn't help. If you set the normal interval to 2 or 3 min and then the retry interval to 1 minute, you will won't know when the server rebooted. Because by the time Nagios checks and both options, the server will have already come back up.

I might be making this harder than it needs to be, but there has to be an easy way just simply get notified when the server reboots itself. No interval checks or certain timed checks, just simply a notification that the server has rebooted. Is there not???

ZiPPy

jamescondron · 06-08-2009, 05:57 PM

How about some simple socket programming, your machine has a socket-server running which, when some specific data is received, flags up a notification. The other machine connects and sends this specific data on boot.

twantrd · 06-09-2009, 02:49 AM

Quote:

I might be making this harder than it needs to be, but there has to be an easy way just simply get notified when the server reboots itself.

Samotnik provided a very easy and quick solution for your problem. Do you know where your startup scripts are? After you know that piece, read up on bash scripting or find a bash script on the web that has an example of sending a piece of email.

-twantrd

ZiPPygEEk · 06-09-2009, 11:48 AM

I don't script, nor do I have time to learn how to script. I am looking for the solution in Nagios. How are other people getting notified of reboots in Nagios? I highly doubt everybody is creating scripts for this type of notification. I'm not trying to be stubborn here, or difficult, but I simply not looking to learn how to script. I just need to get the basic level (at least for now) of Nagios up and running. The only piece I have missing right now, is the notification for restarts.

Thanks,

ZiPPy

jamescondron · 06-09-2009, 11:53 AM

Server Side:

Code:

#!/usr/bin/env python
#
"""
Server Side
Create connection, listen for 'Booted'
Flag
"""

import socket, sys

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = ''
port = 3000
s.bind((host,port))

while(1):
    s.listen(1)
    conn, addr = s.accept()
    
    data = conn.recv( 1000000 )
    if data == "Reboot":
        #Do Something To Notify You
        print "Theres One Reboot"
    
conn.close()

Client Side:

Code:

#!/usr/bin/env python
#
"""
Client Side:
Send the word 'Reboot'
When 'Done' received, exit.
"""

import socket, sys

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = "localhost"
port = 3000
s.connect((host,port))
s.send("Reboot")

i = 0
s.close()

I'm sure you can work out what to change for your set up. There, in simple python, a notifier. I thought scripting/programming ability was more or less a prerequisite of most sysadmin jobs nowadays.

ZiPPygEEk · 06-09-2009, 03:30 PM

First of all, what you have posted isn't exactly a "simple" script in Python. Your dealing with a socket and sys class, which isn't exactly something simple. Secondly, No! NOT ALL sysadmin know how to script or choose to script.

I believe the Unix adage said it best: "do one thing and do it well". I can most definitely say scripting for me is not one of them, or coding for that matter.

Secondly, I appreciate you posting the code but it still doesn't help me. I can't just take some code or script that somebody posted and throw it on a config of mine. More importantly, are we still dealing with Nagios here or something different? Whether it stupidity, hard seeing, I simply don't understand why this is so hard.

Short view:

Server reboots
Nagios sees this
Nagios sends notification of server reboot

How do you configure this in Nagios? Documentation? No ...the documentation simply states what functions, types ect... are, not how to use them.

Once again ...how are people getting notification of system reboots in Nagios? If I get razzed for this post, that's fine, but at least give me some direction on this notification =)

Cheers lads,

ZiPPy

jamescondron · 06-09-2009, 03:40 PM

Do one thing and do it all is a software consideration, your code should do one thing and do it well. Why are you determined to use nagios? Nagios is not the tool for this job. Nagios is the tool for lazy sysadmins who just want pretty screens to show the shareholders; if you want to know when a system boots up, then get the system to tell you. Much the same way I rely on the ringer on my phone to tell me a call is coming in, not a third party.

ZiPPygEEk · 06-09-2009, 04:06 PM

Lazy sysadmins? What are you talking about? Nagios is a great tool to notify me when servers have issues, or network devices are down, overloaded ect... Lazy sysadmins??? Where do you get off saying that? How do you monitor your systems and network? I'm not talking about your little network at home man, this is for an enterprise network. You know what, why am I even bothering typing a response. Your just going to come back with some smart remark, I should of known this would end up to this point after your first ridiculous comment.

And once again ...I search for other Nagios admins. There has to be some out there ...have to be!

Cheers,

ZiPPy

jamescondron · 06-09-2009, 04:18 PM

500 (ish) Windows boxes, about 30 Linux, a file and app server, an on-site web and mail server (with our intranet) and a backup server. Those are ignoring the ones I have no immediate control over. This is at a reasonably well known high street bank, so I have to remotely look after a wide network, with emphasis on security.

Oh, and I think you mean should have known.

Now, back to the point at hand; why are you determined to use Nagios? You have several different solutions presented to you here.