[SOLVED] Apache web server fails to start after Manjaro update

penyuan · 05-17-2017, 04:49 AM

Hello,
This is a problem I've been having trouble getting help for, and hope someone here can help!

I just rebooted my Manjaro LAMP web server after the most recent round of Manjaro Linux updates. Unfortunately, I noticed that the Apache httpd service fails to start with this output:

Quote:

systemctl status httpd
● httpd.service - Apache Web Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset
Active: failed (Result: exit-code) since Tue 2017-05-16 20:39:06 CST; 1min 37
Process: 554 ExecStop=/usr/bin/httpd -k graceful-stop (code=exited, status=0/S
Process: 492 ExecStart=/usr/bin/httpd -k start -DFOREGROUND (code=exited, stat
Main PID: 492 (code=exited, status=1/FAILURE)
May 16 20:39:04 [hostname] systemd[1]: Started Apache Web Server.
May 16 20:39:05 [hostname] systemd[1]: httpd.service: Main process exited, code=ex
May 16 20:39:06 [hostname] httpd[554]: httpd (no pid file) not running
May 16 20:39:06 [hostname] systemd[1]: httpd.service: Unit entered failed state.
May 16 20:39:06 [hostname] systemd[1]: httpd.service: Failed with result 'exit-cod

I've always used httpd in a "set and forget" kind of way and don't really know its inner workings. And I just looked at /var/log/httpd/error_log and saw this at the end:

Quote:

[Tue May 16 20:39:05.654816 2017] [ssl:emerg] [pid 492] AH01903: Failed to configure CA certificate chain!
[Tue May 16 20:39:05.656540 2017] [ssl:emerg] [pid 492] AH02312: Fatal error initialising mod_ssl, exiting.
AH00016: Configuration Failed

So looks like some problem with HTTPS configuration?? I haven't manually changed any settings before the recent update, how do I fix this?? BTW I'm using Let's Encrypt with a CertBot service that runs daily, but it is also failing because it says the httpd service is not running.

Thank you!

Habitual · 05-17-2017, 07:54 AM

Code:

sudo a2enmod mod_ssl

and see what's what.

penyuan · 05-17-2017, 08:18 AM

Quote:

Originally Posted by Habitual

Code:

sudo a2enmod mod_ssl

and see what's what.

Thank you! I managed to find and install a2enmod from the Arch AUR, and running

Code:

sudo a2enmod mod_ssl

gave me this:

Quote:

ERROR: Module mod_ssl does not exist!

How did this happen??? And how do I make mod_ssl re-exist??

EDIT: I see that the file /usr/lib/httpd/modules/mod_ssl.so *exists*, so what's happening???

EDIT 2: Here is the output from a2enmod, which doesn't include mod_ssl:

Quote:

a2enmod
Your choices are: actions alias asis auth_basic auth_digest authn_alias authn_anon authn_dbd authn_dbm authn_default authn_file authnz_ldap authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cern_meta cgi cgid charset_lite dav dav_fs dav_lock dav_svn dbd deflate dir disk_cache dnssd dump_io env expires ext_filter fastcgi fcgid file_cache filter headers ident imagemap include info ldap log_config log_forensic logio mem_cache mime mime_magic mod-security mod_chroot mono negotiation pagespeed perl php5 php5_cgi proxy proxy_ajp proxy_balancer proxy_connect proxy_ftp proxy_http proxy_scgi qos reqtimeout rewrite rpaf setenvif speling ssl status substitute suexec suphp unique_id userdir usertrack version vhost_alias wsgi
Which module(s) do you want to enable (wildcards ok)?

I tried enabling "ssl" but that didn't help...

Habitual · 05-18-2017, 09:33 AM

/var/log/httpd/error_log have any clues?

ondoho · 05-18-2017, 12:27 PM

please rerun "systemctl status apache2" (wait what, why is it httpd on your system?) with full line length, there's a switch for that in systemctl.
also more journalctl output might be available to help with the issue.

penyuan · 05-18-2017, 02:09 PM

Quote:

Originally Posted by ondoho

please rerun "systemctl status apache2" (wait what, why is it httpd on your system?) with full line length, there's a switch for that in systemctl.
also more journalctl output might be available to help with the issue.

OK, I managed to get full line length output for systemctl here:

Quote:

● httpd.service - Apache Web Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-05-19 02:39:30 CST; 20min ago
Process: 8731 ExecStop=/usr/bin/httpd -k graceful-stop (code=exited, status=0/SUCCESS)
Process: 8726 ExecStart=/usr/bin/httpd -k start -DFOREGROUND (code=exited, status=1/FAILURE)
Main PID: 8726 (code=exited, status=1/FAILURE)

May 19 02:39:29 [my hostname] systemd[1]: Started Apache Web Server.
May 19 02:39:30 [my hostname] systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
May 19 02:39:30 [my hostname] httpd[8731]: httpd (no pid file) not running
May 19 02:39:30 [my hostname] systemd[1]: httpd.service: Unit entered failed state.
May 19 02:39:30 [my hostname] systemd[1]: httpd.service: Failed with result 'exit-code'.

Here is the full line length output from journalctl:

Quote:

May 19 02:39:25 [my hostname] polkitd[378]: Registered Authentication Agent for unix-process:8717:19443680 (system bus name : 1.56 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
May 19 02:39:29 [my hostname] polkitd[378]: Operator of unix-process:8717:19443680 successfully authenticated as unix-user:[my username] to gain ONE-SHOT authorization for action org.freedesktop.systemd1.manage-units for system-bus-name::1.57 [systemctl restart httpd] (owned by unix-user:[my username])
May 19 02:39:29 [my hostname] systemd[1]: Started Apache Web Server.
May 19 02:39:29 [my hostname] polkitd[378]: Unregistered Authentication Agent for unix-process:8717:19443680 (system bus name :1.56, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
May 19 02:39:30 [my hostname] systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
May 19 02:39:30 [my hostname] httpd[8731]: httpd (no pid file) not running
May 19 02:39:30 [my hostname] systemd[1]: httpd.service: Unit entered failed state.
May 19 02:39:30 [my hostname] systemd[1]: httpd.service: Failed with result 'exit-code'.

And once again, here is the relevant output from /var/log/httpd/error_log:

Quote:

[Fri May 19 02:39:30.036834 2017] [ssl:emerg] [pid 8726] AH01903: Failed to configure CA certificate chain!
[Fri May 19 02:39:30.037539 2017] [ssl:emerg] [pid 8726] AH02312: Fatal error initialising mod_ssl, exiting.
AH00016: Configuration Failed

I don't know why the service is called "httpd" instead of something else...

Does this provide any clues??? Thanks for your help!!

EDIT: Oh, and here's the "sudo a2enmod mod_ssl" output again:

Quote:

ERROR: Module mod_ssl does not exist!

But the file mod_ssl.so does exist.

Habitual · 05-18-2017, 03:05 PM

Is mod_ssl the only error you see?
Because I seen 2 distinct errors that may have a common denominator
...ca-certificates...

Code:

Description: Common CA certificates
 This package includes PEM files of CA certificates to allow SSL-based
 applications to check for the authenticity of SSL connections.

Hopefully, that helps.

penyuan · 05-18-2017, 03:10 PM

Quote:

Originally Posted by Habitual

Is mod_ssl the only error you see?
Because I seen 2 distinct errors that may have a common denominator
...ca-certificates...

Code:

Description: Common CA certificates
 This package includes PEM files of CA certificates to allow SSL-based
 applications to check for the authenticity of SSL connections.

Hopefully, that helps.

Thanks for your quick response!!

I just ran "sudo pacman -S ca-certificates" to re-install the ca-certificates package (it was already installed). Unfortunately running "systemctl restart httpd" still didn't work...

And yes, mod_ssl missing is the only error when I run "sudo a2enmod mod_ssl"...

ondoho · 05-19-2017, 10:26 AM

please run this:

Code:

/usr/bin/httpd -k start -DFOREGROUND

in a terminal, as root, and post what output you get.
actually sorry, no, you have to add more verbosity.
please read

Code:

man httpd

then add more verbosity to the above command.
post the output here.

penyuan · 05-19-2017, 09:05 PM

Quote:

Originally Posted by ondoho

please run this:

Code:

/usr/bin/httpd -k start -DFOREGROUND

in a terminal, as root, and post what output you get.
actually sorry, no, you have to add more verbosity.
please read

Code:

man httpd

then add more verbosity to the above command.
post the output here.

OK. With it I got a new clue, here is the output of the command:

Quote:

AH00526: Syntax error on line 535 of /etc/httpd/conf/httpd.conf:
SSLCertificateKeyFile: file '/etc/letsencrypt/live/[my domain]/privkey.pem' does not exist or is empty

I checked '/etc/letsencrypt/live/[my domain]/privkey.pem', and it definitely exists, is not empty, and I definitely did not change it before or after the problems started.
What can I do?? Thanks!

ondoho · 05-20-2017, 02:03 AM

Quote:

Originally Posted by penyuan

Code:

AH00526: Syntax error on line 535 of /etc/httpd/conf/httpd.conf:
SSLCertificateKeyFile: file '/etc/letsencrypt/live/[my domain]/privkey.pem' does not exist or is empty

What can I do??

so this is now a letsencrypt issue.
no help from me there, sorry.

bathory · 05-20-2017, 03:27 AM

Hi

Quote:

AH00526: Syntax error on line 535 of /etc/httpd/conf/httpd.conf:
SSLCertificateKeyFile: file '/etc/letsencrypt/live/[my domain]/privkey.pem' does not exist or is empty
I checked '/etc/letsencrypt/live/[my domain]/privkey.pem', and it definitely exists, is not empty, and I definitely did not change it before or after the problems started.
What can I do?? Thanks!

Take a look a this post at letsencrypt forum

Regards

penyuan · 05-20-2017, 04:42 AM

Quote:

Originally Posted by ondoho

so this is now a letsencrypt issue.
no help from me there, sorry.

I followed the link by @bathory and was able to resolve this issue, but it led to another. Please see below.

Quote:

Originally Posted by bathory

Hi

Take a look a this post at letsencrypt forum

Regards

Thank you for this. Following the suggestion at the link solved that particular error message but now there is a new one:

Quote:

(13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs

I have no idea what this means... Can you still help??

bathory · 05-20-2017, 08:01 AM

Quote:

(13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs

Most likely apache is still running when you try to start it again. Stop it and use the following command to see if that's the case:

Code:

netstat -tanpl|grep 80

You can also use "kill" to kill the running httpd process(es).
Then try to start it again.

Could be that you define 2 times "port 80" in the configuration files. Check the occurrences of

Code:

Listen 80

and make sure there is only one.

penyuan · 05-20-2017, 12:55 PM

Quote:

Originally Posted by bathory

Most likely apache is still running when you try to start it again. Stop it and use the following command to see if that's the case:

Code:

netstat -tanpl|grep 80

You can also use "kill" to kill the running httpd process(es).
Then try to start it again.

Could be that you define 2 times "port 80" in the configuration files. Check the occurrences of

Code:

Listen 80

and make sure there is only one.

OK, I did:

ran "systemctl stop httpd"
"sudo netstat -tanpl|grep 80" showed nothing (so I guessed httpd was really stopped)
checked "htop" and "sudo htop" for httpd or apache and nothing was listed
checked my httpd.conf configuration file to make sure there's only one "Listen 80"
did "systemctl start httpd"
It fails with the same errors as before...
Tried to reboot server with same result...

Anything else I can try??? Sorry about being difficult but I appreciate your help!