Apache webserver on CentOS7 hanging from processes
systemctl status httpd shows about 100 of these and still shows running, although sites are timing out. restarting the server clears it but I don't want to do that nightly. tried just restarting apache but was taking forever.
CGroup: /system.slice/httpd.service ├─ 8499 /usr/sbin/httpd -DFOREGROUND ├─ 8788 /usr/sbin/httpd -DFOREGROUND ├─11803 /usr/sbin/httpd -DFOREGROUND ├─12028 /usr/sbin/httpd -DFOREGROUND ├─12068 /usr/sbin/httpd -DFOREGROUND ├─12077 /usr/sbin/httpd -DFOREGROUND ├─12079 /usr/sbin/httpd -DFOREGROUND ├─12080 /usr/sbin/httpd -DFOREGROUND ├─12087 /usr/sbin/httpd -DFOREGROUND |
We need more information to help, I think.
What do you mean by restarting apache "taking forever"? How long is "forever"? How many sites are on the server? When things are timing out, what does Code:
netstat -tnp | grep 80 |
The server is serving 12 sites. We only have 1 external IP pointing to the server. I let it try restarting apache for over a minute until canceling and rebooting. Not sure what netstat shows, I rebooted before checking. I'll do that if it happens again. It was working fine until that point and after rebooting.
There was an external server that had a website mounted in /etc/fstab that went down. Possibly that? Currently netstat shows: [root@webserver]# netstat -tpn | grep 443 tcp 0 0 dmz.ip.address:47868 external.ip.address:443 TIME_WAIT - tcp 0 0 dmz.ip.address:47876 external.ip.address:443 TIME_WAIT - tcp 0 0 dmz.ip.address:47855 external.ip.address:443 TIME_WAIT - tcp 0 0 dmz.ip.address:47822 external.ip.address:443 TIME_WAIT - tcp 0 0 dmz.ip.address:47834 external.ip.address:443 TIME_WAIT - tcp 0 0 dmz.ip.address:47866 external.ip.address:443 TIME_WAIT - tcp 0 0 dmz.ip.address:47878 external.ip.address:443 TIME_WAIT - tcp 0 0 dmz.ip.address:47833 external.ip.address:443 TIME_WAIT - [root@webserver]# netstat -tpn | grep 80 tcp 0 0 dmz.ip.address:80 external.ip.address:57307 SYN_RECV - tcp 0 0 storage.ip.address:807 storage.ip.address:2049 ESTABLISHED - |
Quote:
Also, look in the httpd logs to see who's requesting what from the server at the time of the slowdown/freeze. |
All the sites that are served from the webserver are mounted on the webserver from somewhere else. Mostly from a SAN but also from a couple other servers.
[Tue Jul 09 14:49:41.111659 2019] [mpm_prefork:notice] [pid 2418] AH00170: caught SIGWINCH, shutting down gracefully [Tue Jul 09 14:49:42.279250 2019] [suexec:notice] [pid 2741] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Tue Jul 09 14:49:42.286711 2019] [ssl:warn] [pid 2741] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366) [Tue Jul 09 14:49:42.331639 2019] [auth_digest:notice] [pid 2741] AH01757: generating secret for digest authentication ... [Tue Jul 09 14:49:42.332197 2019] [lbmethod_heartbeat:notice] [pid 2741] AH02282: No slotmem from mod_heartmonitor [Tue Jul 09 14:49:42.339161 2019] [ssl:warn] [pid 2741] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366) [Tue Jul 09 14:49:42.368219 2019] [mpm_prefork:notice] [pid 2741] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 SVN/1.7.14 PHP/7.2.19 configured -- resuming normal operations [Tue Jul 09 14:49:42.368250 2019] [core:notice] [pid 2741] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' Those errors and php errors are the only ones in the error_logs |
Activity is in the access_logs, not the error_logs. That would show you who is requesting what. Look specifically at/around the time of the slowdown/freeze.
Given that the content is remote to the web server, I'd suspect a communication issue there. The logs will help narrow that down. A problem in that communication could also account for the slowness of a restart. |
All times are GMT -5. The time now is 08:40 AM. |