I have a nagios install on CentOS that has been working away fine for the last year and a half. Today I login successfully however get the Internal Server Error and can't navigate anywhere. Same for nconf which is also installed

I've looked through past threads here on the issue, most seem to be from fresh installs, SELinux is already disabled.

Running /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

throws no errors. Nagios.log looks like its still running ok checks are still being made.

I'm a bit stumped here tbh, any help would be appreciated

When you visit the page, what kind of errors do you see in the web server logs?

in the httpd error_log I see the below


[Mon Jul 21 12:22:28 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jul 21 12:22:28 2014] [notice] Digest: generating secret for digest authentication ...
[Mon Jul 21 12:22:28 2014] [notice] Digest: done
[Mon Jul 21 12:22:28 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Mon Jul 21 12:27:24 2014] [notice] caught SIGTERM, shutting down
[Mon Jul 21 12:27:25 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jul 21 12:27:25 2014] [notice] Digest: generating secret for digest authentication ...
[Mon Jul 21 12:27:25 2014] [notice] Digest: done
[Mon Jul 21 12:27:25 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Mon Jul 21 12:48:04 2014] [notice] caught SIGTERM, shutting down
[Mon Jul 21 12:48:50 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Jul 21 12:48:51 2014] [notice] Digest: generating secret for digest authentication ...
[Mon Jul 21 12:48:51 2014] [notice] Digest: done

Hello,

Can you please disable the suEXEC and try again and if you are still facing same then re install it as followed

Step 1 - Install Packages on Monitoring Server

rpm -Uvh http://dl.fedoraproject.org/pub/epel...6-8.noarch.rpm
rpm -Uvh http://rpms.famillecollet.com/enterp...-release-6.rpm
yum -y install nagios nagios-plugins-all nagios-plugins-nrpe nrpe php httpd

chkconfig httpd on && chkconfig nagios on
service httpd start && service nagios start

We should also enable SWAP memory on this droplet, at least 2GB:

dd if=/dev/zero of=/swap bs=1024 count=2097152
mkswap /swap && chown root. /swap && chmod 0600 /swap && swapon /swap
echo /swap swap swap defaults 0 0 >> /etc/fstab
echo vm.swappiness = 0 >> /etc/sysctl.conf && sysctl -p

Step 2 - Set Password Protection

Set Nagios Admin Panel Password:

htpasswd -c /etc/nagios/passwd nagiosadmin

Make sure to keep this username as "nagiosadmin" - otherwise you would have to change /etc/nagios/cgi.cfg and redefine authorized admin.

Now you can navigate over to your droplet's IP address http://IP/nagios and login.



Step 4 -Install NRPE on Clients

rpm -Uvh http://dl.fedoraproject.org/pub/epel...6-8.noarch.rpm
rpm -Uvh http://rpms.famillecollet.com/enterp...-release-6.rpm
yum -y install nagios nagios-plugins-all nrpe
chkconfig nrpe on
This next step is where you get to specify any manual commands that Monitoring server can send via NRPE to these client hosts.

Make sure to change allowed_hosts to your own values.

Edit /etc/nagios/nrpe.cfg

log_facility=daemon
pid_file=/var/run/nrpe/nrpe.pid
server_port=5666
nrpe_user=nrpe
nrpe_group=nrpe
allowed_hosts=198.211.117.251
dont_blame_nrpe=1
debug=0
command_timeout=60
connection_timeout=300
include_dir=/etc/nrpe.d/
command[check_users]=/usr/lib64/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib64/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_disk]=/usr/lib64/nagios/plugins/check_disk -w 20% -c 10% -p /dev/vda
command[check_zombie_procs]=/usr/lib64/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib64/nagios/plugins/check_procs -w 150 -c 200
command[check_procs]=/usr/lib64/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $A

Note:

In check_disk above, the partition being checked is /dev/vda - make sure your droplet has the same partition by running df -h / You can also modify when to trigger warnings or critical alerts - above configuration sets Warning at 20% free disk space remaining, and Critical alert at 10% free space remaining.

We should also setup firewall rules to allow connections from our Monitoring server to those clients and drop everyone else:

iptables -N NRPE
iptables -I INPUT -s 0/0 -p tcp --dport 5666 -j NRPE
iptables -I NRPE -s 198.211.117.251 -j ACCEPT
iptables -A NRPE -s 0/0 -j DROP
/etc/init.d/iptables save

Now you can start NRPE on all of your client hosts:

service nrpe start

Step 5 - Add Server Configurations on Monitoring Server

Back on our Monitoring server, we will have to create config files for each of our client servers:

echo "cfg_dir=/etc/nagios/servers" >> /etc/nagios/nagios.cfg
cd /etc/nagios/servers
touch cloudmail.tk.cfg
touch emailocean.tk.cfg
Edit each client's configuration file and define which services you would like monitored.

nano /etc/nagios/servers/cloudmail.tk.cfg
Add the following lines:

define host {
use linux-server
host_name cloudmail.tk
alias cloudmail.tk
address 198.211.107.218
}

define service {
use generic-service
host_name cloudmail.tk
service_description PING
check_command check_ping!100.0,20%!500.0,60%
}

define service {
use generic-service
host_name cloudmail.tk
service_description SSH
check_command check_ssh
notifications_enabled 0
}

define service {
use generic-service
host_name cloudmail.tk
service_description Current Load
check_command check_local_load!5.0,4.0,3.0!10.0,6.0,4.0
}
You can add more services to be monitored as desired. Same configuration should be added for second client, emailocean.tk, with different IP address and host_name:

This is a snippet of /etc/nagios/servers/emailocean.tk.cfg:

define host {
use linux-server
host_name emailocean.tk
alias emailocean.tk
address 198.211.112.99
}

...
You can add additional clients to be monitored as /etc/nagios/servers/AnotherHostName.cfg

Finally, after you are done adding all the client configurations, you should set folder permissions correctly and restart Nagios on your Monitoring Server:

chown -R nagios. /etc/nagios
service nagios restart
Step 6 - Monitor Hosts in Nagios
Navigate over to your Monitoring Server's IP address http://IP/nagios and enter password set in Step 2.

---
linuxfreakz

Hi Thanks for the reply. I tried disabling suexec again but that didnt work.
I'm reluctant to try reinstalling yet because as I said this is an install that has been working without issue for the last year so I would rather find what caused the issue than start from scratch again. A lot of work went into getting it setup as far as this.
Strange thing is that nagios appears to be running correctly, it is still sending problem and recovery alerts to the correct email addreses its just I cannot access the web interface, I also cannot access the web interface for nconf or vshell either.
As this is a vm I might be able to revert to a snapshot from last week when I know all was working ok
The only errors in the httpd error_log is related to favicon.ico but I don't believe this is the issue as I placed the file into the correct dir and the problem is still there

[Mon Jul 21 17:14:55 2014] [error] [client ] File does not exist: /var/www/html/favicon.ico
[Tue Jul 22 04:25:28 2014] [error] [client ] File does not exist: /var/www/html/favicon.ico
[Tue Jul 22 04:36:50 2014] [notice] caught SIGTERM, shutting down
[Tue Jul 22 04:37:55 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Jul 22 04:37:55 2014] [notice] Digest: generating secret for digest authentication ...
[Tue Jul 22 04:37:55 2014] [notice] Digest: done
[Tue Jul 22 04:37:55 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
[Tue Jul 22 05:24:11 2014] [notice] caught SIGTERM, shutting down
[Tue Jul 22 05:25:07 2014] [notice] Digest: generating secret for digest authentication ...
[Tue Jul 22 05:25:07 2014] [notice] Digest: done
[Tue Jul 22 05:25:07 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations

Discovered what the problem was here. Someone else had changed the ip address of the server used for LDAP authentication to the nagios host so obviously the website wouldnt load when username and password were entered.