systemctl status httpd shows about 100 of these and still shows running, although sites are timing out. restarting the server clears it but I don't want to do that nightly. tried just restarting apache but was taking forever.

CGroup: /system.slice/httpd.service
├─ 8499 /usr/sbin/httpd -DFOREGROUND
├─ 8788 /usr/sbin/httpd -DFOREGROUND
├─11803 /usr/sbin/httpd -DFOREGROUND
├─12028 /usr/sbin/httpd -DFOREGROUND
├─12068 /usr/sbin/httpd -DFOREGROUND
├─12077 /usr/sbin/httpd -DFOREGROUND
├─12079 /usr/sbin/httpd -DFOREGROUND
├─12080 /usr/sbin/httpd -DFOREGROUND
├─12087 /usr/sbin/httpd -DFOREGROUND

We need more information to help, I think.
What do you mean by restarting apache "taking forever"? How long is "forever"?
How many sites are on the server?
When things are timing out, what does show? (grep for 443 instead if sites are using https) That is, how many IPs are connected to the web server?

The server is serving 12 sites. We only have 1 external IP pointing to the server. I let it try restarting apache for over a minute until canceling and rebooting. Not sure what netstat shows, I rebooted before checking. I'll do that if it happens again. It was working fine until that point and after rebooting.

There was an external server that had a website mounted in /etc/fstab that went down. Possibly that?

Currently netstat shows:
[root@webserver]# netstat -tpn | grep 443
tcp 0 0 dmz.ip.address:47868 external.ip.address:443 TIME_WAIT -
tcp 0 0 dmz.ip.address:47876 external.ip.address:443 TIME_WAIT -
tcp 0 0 dmz.ip.address:47855 external.ip.address:443 TIME_WAIT -
tcp 0 0 dmz.ip.address:47822 external.ip.address:443 TIME_WAIT -
tcp 0 0 dmz.ip.address:47834 external.ip.address:443 TIME_WAIT -
tcp 0 0 dmz.ip.address:47866 external.ip.address:443 TIME_WAIT -
tcp 0 0 dmz.ip.address:47878 external.ip.address:443 TIME_WAIT -
tcp 0 0 dmz.ip.address:47833 external.ip.address:443 TIME_WAIT -

[root@webserver]# netstat -tpn | grep 80
tcp 0 0 dmz.ip.address:80 external.ip.address:57307 SYN_RECV -
tcp 0 0 storage.ip.address:807 storage.ip.address:2049 ESTABLISHED -

What? I don't understand.

Also, look in the httpd logs to see who's requesting what from the server at the time of the slowdown/freeze.

All the sites that are served from the webserver are mounted on the webserver from somewhere else. Mostly from a SAN but also from a couple other servers.

[Tue Jul 09 14:49:41.111659 2019] [mpm_prefork:notice] [pid 2418] AH00170: caught SIGWINCH, shutting down gracefully
[Tue Jul 09 14:49:42.279250 2019] [suexec:notice] [pid 2741] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Jul 09 14:49:42.286711 2019] [ssl:warn] [pid 2741] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Jul 09 14:49:42.331639 2019] [auth_digest:notice] [pid 2741] AH01757: generating secret for digest authentication ...
[Tue Jul 09 14:49:42.332197 2019] [lbmethod_heartbeat:notice] [pid 2741] AH02282: No slotmem from mod_heartmonitor
[Tue Jul 09 14:49:42.339161 2019] [ssl:warn] [pid 2741] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Jul 09 14:49:42.368219 2019] [mpm_prefork:notice] [pid 2741] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 SVN/1.7.14 PHP/7.2.19 configured -- resuming normal operations
[Tue Jul 09 14:49:42.368250 2019] [core:notice] [pid 2741] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Those errors and php errors are the only ones in the error_logs

Activity is in the access_logs, not the error_logs. That would show you who is requesting what. Look specifically at/around the time of the slowdown/freeze.

Given that the content is remote to the web server, I'd suspect a communication issue there. The logs will help narrow that down.

A problem in that communication could also account for the slowness of a restart.

1 members found this post helpful.