Apache, No space left on device, Couldn't create the proxy mutex

bigmen58 · 12-16-2013, 07:11 AM

Hello,

I have Apache 2.4.6 http server and sometimes after 7 - 15 days from last reboot Apache crashing. When i'm trying start httpd in error_log i have this:

Quote:

[Mon Dec 14 00:54:15.131973 2013] [lbmethod_heartbeat:notice] [pid 871455:tid 140283000383424] AH02282: No slotmem from mod_heartmonitor
[Mon Dec 14 00:54:15.133058 2013] [core:emerg] [pid 871455:tid 140283000383424] (28)No space left on device: AH00023: Couldn't create the fcgid-pipe mutex
[Mon Dec 14 00:55:01.876934 2013] [:notice] [pid 871508:tid 140395925907392] mod_hostinglimits: version 1.0-7. LVE mechanism enabled
[Mon Dec 14 00:55:01.877007 2013] [:notice] [pid 871508:tid 140395925907392] mod_hostinglimits: found apr extention version 2
[Mon Dec 14 00:55:01.877011 2013] [:notice] [pid 871508:tid 140395925907392] mod_hostinglimits: apr_lve_environment_init_group check ok
[Mon Dec 14 00:55:02.000159 2013] [core:emerg] [pid 871508:tid 140395925907392] (28)No space left on device: AH00023: Couldn't create the proxy mutex
[Mon Dec 14 00:55:02.000189 2013] [proxy:crit] [pid 871508:tid 140395925907392] (28)No space left on device: AH02478: failed to create proxy mutex
AH00016: Configuration Failed

I found somewhere that Apache don't release semaphores. Then resolve is manually remove opened semaphores:

Quote:

ipcs | grep apache | awk '{print $2}' > sem.txt
for i in `cat sem.txt`; do { ipcrm -s $i; }; done;

After that httpd started.

I've noticed that after httpd restart "ipcs | grep apache | wc -l" return about 21 items. Before restart, after httpd crash same command return 125.

How prevent it in future? Why it happens?

pan64 · 12-16-2013, 07:49 AM

I do not really understand you. No space left on device means disk full. Have you checked your drives? (df -h)

bigmen58 · 12-16-2013, 07:55 AM

Quote:

Originally Posted by pan64

I do not really understand you. No space left on device means disk full. Have you checked your drives? (df -h)

Yes, all paritions have free space.

I think "No space left on device" refers to semaphores and shared memory limits also, not only hard drive space.

A few minutes ago i was increased kernel.sem to:

kernel.sem = 500 64000 64 1024 (before: 250 32000 32 128)

Maybe it would help, but i don't know why apache use more than 100 semaphores.

Any ideas?

grim76 · 12-17-2013, 08:59 AM

You could have plenty of space on a system and be out of inodes. I have seen that happen before.

I would run a df -i and see if you are out of inodes on any of your volumes.

bigmen58 · 12-17-2013, 03:18 PM

Yeah i know about inodes - in my case all partitions have a lot of free inodes.

As I said, I changed kernel.sem limit, and I'm waiting - It's OK, for the time being.

Jan-E · 09-26-2014, 08:43 AM

@bigmen58: I had the same. Apache 2.4.10 on Centos 6 would not restart after an uptime of 448 days. There were 124 semaphores in use. After releasing them only 8 were left and Apache started once again. Thanks for the solution.

GaWdLy · 09-27-2014, 01:16 AM

The updated sysctl settings should help with the problem.