Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
This has been happening for a while now and I can't seem to isolate the issue. Apache will run fine for anywhere from a few days to a few weeks but then all of a sudden it will "crash". I put that in quotes because the process is not completely crashing. Basically when this happens, Apache just refuses to accept any connections and cannot be restarted using the init script (it simply fails to stop then fails to start and gives an error about the address being in use). Every time this happens, I just do a "killall httpd", then start the process with the init script again and everything works perfectly. Of course having this happen is quite a nuisance.
Does anyone know what could be going on here? I have checked the error log and it only contains notices, no errors.
Ok I did that. But in the mean time, is it possible that it is reaching the max connections or would that have shown up in my error log? I had the log level set to warn.
I can't seem to get any info from the error logs. My error logs for the past 8+ days are non existent and before that, I do not see any useful information. Here is the message I get when I try to restart apache after a "crash".
[root@srv1 httpd]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]
[root@srv1 httpd]#
[root@srv1 httpd]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]
[root@srv1 httpd]#
May be your apache didn't stop properly and it makes you to kill all the httpd processes to restart it. I had the same issue with my client's server, and the reason for that was the server reached maximum clients and got hanged. You can check them with your error logs, while analyzing I found that developer made a mistake while developing a ffmpeg script and that ffmpeg process forms a loop and multiplies the number of httpd requests to the server which completely eats up the number of connections.
I'm still having this issue. I have tried everything I can think of, including changing the apache LogLevel to debug. Even in debug, there is nothing in the error log or system messages log around the time apache "crashes".
Here is my apache error log with debug turned on. Keep in mind that I have no idea if these messages are from around the time of the crash because I do not know when it crashed.
Code:
[Sun Nov 27 04:02:14 2011] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 27 04:02:14 2011] [notice] Digest: done
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain1.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain2.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain3.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain4.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain5.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain6.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain7.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain8.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain9.com
[Sun Nov 27 04:02:14 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8305470 rmm=0x83054a0 for VHOST: domain10.com
[Sun Nov 27 04:02:14 2011] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Sun Nov 27 04:02:14 2011] [info] LDAP: SSL support available
[Sun Nov 27 04:02:14 2011] [info] mod_fcgid: Process manager 2357 started
[Sun Nov 27 04:02:14 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Nov 27 04:02:14 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Sun Nov 27 04:02:15 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Sun Nov 27 04:02:15 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(374): shmcb_init allocated 512000 bytes of shared memory
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(554): entered shmcb_init_memory()
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(576): for 512000 bytes, recommending 4266 indexes
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(619): shmcb_init_memory choices follow
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(621): division_mask = 0x1F
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(623): division_offset = 64
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(625): division_size = 15998
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(627): queue_size = 1604
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(629): index_num = 133
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(631): index_offset = 8
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(633): index_size = 12
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(635): cache_data_offset = 8
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(637): cache_data_size = 14386
[Sun Nov 27 04:02:15 2011] [debug] ssl_scache_shmcb.c(650): leaving shmcb_init_memory()
[Sun Nov 27 04:02:15 2011] [info] Shared memory session cache initialised
[Sun Nov 27 04:02:15 2011] [info] Init: Initializing (virtual) servers for SSL
[Sun Nov 27 04:02:15 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2367 for worker proxy:reverse
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2367 for (*)
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2368 for worker proxy:reverse
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2368 for (*)
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2369 for worker proxy:reverse
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2369 for (*)
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2370 for worker proxy:reverse
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2370 for (*)
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2373 for worker proxy:reverse
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:15 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2373 for (*)
[Sun Nov 27 04:02:15 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 27 04:02:15 2011] [info] Server built: Sep 7 2011 21:44:18
[Sun Nov 27 04:02:15 2011] [debug] prefork.c(996): AcceptMutex: sysvsem (default: sysvsem)
[Sun Nov 27 04:02:15 2011] [notice] Graceful restart requested, doing restart
[Sun Nov 27 04:02:16 2011] [info] mod_fcgid: Process manager 2357 stopped
[Sun Nov 27 04:02:17 2011] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 27 04:02:17 2011] [notice] Digest: done
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain1.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain2.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain3.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain4.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain5.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain6.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain7.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain8.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain9.com
[Sun Nov 27 04:02:17 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8313a20 rmm=0x8313a50 for VHOST: domain10.com
[Sun Nov 27 04:02:17 2011] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Sun Nov 27 04:02:17 2011] [info] LDAP: SSL support available
[Sun Nov 27 04:02:17 2011] [info] mod_fcgid: Process manager 2414 started
[Sun Nov 27 04:02:17 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Nov 27 04:02:17 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Sun Nov 27 04:02:17 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Sun Nov 27 04:02:17 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(374): shmcb_init allocated 512000 bytes of shared memory
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(554): entered shmcb_init_memory()
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(576): for 512000 bytes, recommending 4266 indexes
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(619): shmcb_init_memory choices follow
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(621): division_mask = 0x1F
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(623): division_offset = 64
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(625): division_size = 15998
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(627): queue_size = 1604
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(629): index_num = 133
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(631): index_offset = 8
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(633): index_size = 12
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(635): cache_data_offset = 8
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(637): cache_data_size = 14386
[Sun Nov 27 04:02:17 2011] [debug] ssl_scache_shmcb.c(650): leaving shmcb_init_memory()
[Sun Nov 27 04:02:17 2011] [info] Shared memory session cache initialised
[Sun Nov 27 04:02:17 2011] [info] Init: Initializing (virtual) servers for SSL
[Sun Nov 27 04:02:17 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Sun Nov 27 04:02:17 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 27 04:02:17 2011] [info] Server built: Sep 7 2011 21:44:18
[Sun Nov 27 04:02:17 2011] [debug] prefork.c(996): AcceptMutex: sysvsem (default: sysvsem)
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2430 for worker proxy:reverse
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2430 for (*)
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2431 for worker proxy:reverse
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2431 for (*)
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2432 for worker proxy:reverse
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2432 for (*)
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2433 for worker proxy:reverse
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2433 for (*)
[Sun Nov 27 04:02:17 2011] [notice] Graceful restart requested, doing restart
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2434 for worker proxy:reverse
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:17 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2434 for (*)
[Sun Nov 27 04:02:19 2011] [info] mod_fcgid: Process manager 2414 stopped
[Sun Nov 27 04:02:19 2011] [notice] Digest: generating secret for digest authentication ...
[Sun Nov 27 04:02:19 2011] [notice] Digest: done
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain1.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain2.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain3.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain4.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain5.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain6.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain7.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain8.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain9.com
[Sun Nov 27 04:02:19 2011] [debug] util_ldap.c(2052): LDAP merging Shared Cache conf: shm=0x8328d00 rmm=0x8328d30 for VHOST: domain10.com
[Sun Nov 27 04:02:19 2011] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Sun Nov 27 04:02:19 2011] [info] LDAP: SSL support available
[Sun Nov 27 04:02:19 2011] [info] mod_fcgid: Process manager 2460 started
[Sun Nov 27 04:02:19 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Sun Nov 27 04:02:19 2011] [info] Init: Seeding PRNG with 256 bytes of entropy
[Sun Nov 27 04:02:19 2011] [info] Init: Generating temporary RSA private keys (512/1024 bits)
[Sun Nov 27 04:02:19 2011] [info] Init: Generating temporary DH parameters (512/1024 bits)
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(374): shmcb_init allocated 512000 bytes of shared memory
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(554): entered shmcb_init_memory()
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(576): for 512000 bytes, recommending 4266 indexes
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(619): shmcb_init_memory choices follow
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(621): division_mask = 0x1F
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(623): division_offset = 64
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(625): division_size = 15998
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(627): queue_size = 1604
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(629): index_num = 133
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(631): index_offset = 8
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(633): index_size = 12
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(635): cache_data_offset = 8
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(637): cache_data_size = 14386
[Sun Nov 27 04:02:19 2011] [debug] ssl_scache_shmcb.c(650): leaving shmcb_init_memory()
[Sun Nov 27 04:02:19 2011] [info] Shared memory session cache initialised
[Sun Nov 27 04:02:19 2011] [info] Init: Initializing (virtual) servers for SSL
[Sun Nov 27 04:02:19 2011] [info] Server: Apache/2.2.3, Interface: mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2462 for worker proxy:reverse
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2462 for (*)
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2463 for worker proxy:reverse
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2463 for (*)
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2464 for worker proxy:reverse
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2464 for (*)
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2465 for worker proxy:reverse
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2465 for (*)
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1817): proxy: grabbed scoreboard slot 0 in child 2466 for worker proxy:reverse
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1836): proxy: worker proxy:reverse already initialized
[Sun Nov 27 04:02:19 2011] [debug] proxy_util.c(1930): proxy: initialized single connection worker 0 in child 2466 for (*)
[Sun Nov 27 04:02:19 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Sun Nov 27 04:02:19 2011] [info] Server built: Sep 7 2011 21:44:18
[Sun Nov 27 04:02:19 2011] [debug] prefork.c(996): AcceptMutex: sysvsem (default: sysvsem)
Here is the output of that netstat command while apache is crashed
Crashed is not the same as "no longer accepting connections"
Try running strace -p 2460
you might also try lsof |grep "^httpd"
But also consider that it is not stopping properly with the init script after a "crash"
Here is some output from strace (not sure what this does, but it seemed to be never ending so these are the last few lines of output if that helps at all). From what I could tell, it was almost the same thing repeated over and over again.
I don't have an answer, I don't know centos. I would take that POLLHUP info and search the apache maillist.
Are you doing a graceful shutdown? If I recall the graceful shutdown waits until some conditions are met.
Possibly some sockets are misbehaving.
I see you have selinux too, wheter or not it's a culprit here I don't know. It can get in the way sometimes.
Is it possible that apache is doing graceful restarts on its own and maybe the conditions are not being met so the process becomes halted? In that case wouldn't it at least log the request?
All I really know is that I started having this issue about a year ago and cannot for the life of me figure out what is causing it. I've been using apache for a number of years, and every other issue I've had has been indicated in the log and relatively straightforward to resolve.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.